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Plain language summary 


No Excuses charter schools associated with greater gains in math and literacy 
than traditional public schools 


The No Excuses charter school model focuses heavily on high academic expectations, rigid and 
consistent discipline, extended instructional time, intensive teacher training, and increased 
parental involvement. This review examines the effect of No Excuses charter schools on math and 
literacy achievement. On average, No Excuses charter schools are associated with greater student 
gains on standardized measures of math and literacy achievement when compared to traditional 
public schools - with higher gains for math. 


What did the review study? 


Students from low socio-economic backgrounds or traditionally disadvantaged groups often 
underperform their peers on standardized tests of math and literacy achievement. This 
“achievement gap” is associated with negative education and career outcomes - both short-term 
and long-term. No Excuses charter schools are intended to reduce this gap and improve the 
achievement gains of traditionally disenfranchised groups. 


For the purposes of this review, No Excuses charter schools are those charter schools that highlight 
the importance of high academic expectations for all students, rigid discipline enforcement, 
extended time in school, intensive teacher training, and parental involvement. 


This review examines whether No Excuses charter schools are associated with greater achievement 
gains in math and literacy compared to the achievement gains of similar students enrolled in 
traditional public schools. 


What studies are included? 


This review includes studies that evaluate the effects of No Excuses charter schools on students’ 
literacy and math achievement gains. It includes 18 studies conducted in the United States 
spanning from 1990-2015. 
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What is the aim of this review? 


This Campbell systematic review examines the effects of No Excuses charter schools on 
students’ math and literacy achievement gains compared to similar students in public 
schools. The review summarizes evidence from 18 studies, including 5 randomized 
controlled trials and 13 quasi-experimental studies. 


What are the main results of this review? 


No Excuses charter schools, on average, produced larger math and literacy achievement gains for 
their students than their public school peers - with higher gains for math. These benefits increase 
for three years, at which point the achievement gains stabilized or returned to lower gains. 


What do the findings of this review mean? 


No Excuses charter schools may help underperforming students make larger achievement gains in 
math and literacy, more so than at traditional public schools. Additional research is needed to 
examine the effects of this intervention while accounting for significant issues related to sample 
bias and to determine the scalability of the model. 


What are the limitations in interpretation of this review? 


Readers should take caution when interpreting this review due to several limitations. First, studies 
were inconsistent in reporting (or failed to report) important sample characteristics such as 
percentage of students receiving special education services, number of students exiting or expelled 
after enrollment etc., severely limiting interpretation of the equality of comparison groups and 
factors impacting differences in outcomes. 


Second, follow-up years suggest a sustained significant effect on both math and literacy 
achievement. However, these results must be interpreted with caution as the number of samples 
decreased substantially at follow-up years with just one study accounting for the majority of the 
samples. 


Third, studies included in this review did not report data indicating the degree to which each 
characteristic of No Excuses charter schools was implemented, prohibiting the interpretation of the 
presence, absence, or dosage of the individual characteristics of No Excuses charter schools. 


Fourth, only five of the 18 included studies employed random assignment, limiting interpretation 
of the equality of the comparison groups due to the small number in independent samples using 
random assignment. 
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Fifth, a large number of the included studies examine the effects of the schools of a specific charter 
Management organization: Knowledge is Power Program (KIPP). ‘Though informative, the results 
of this review cannot be considered generalizable to all No Excuses charter schools. 


Finally, most studies measured effects with only one, annual standardized achievement measure. 


How up-to-date is this review? 


The review authors searched for studies through J une 2016. 
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Executive summary/ Abstract 


BACKGROUND 


In response to the school accountability movement and the move toward market-oriented 
education policies in recent decades, many schools and districts have felt increased pressure to find 
effective ways to boost student achievement. However, many districts—largely those serving low- 
income black and hispanic students in urban settings—have still struggled to make sufficient 
academic gains (Barton & Coley, 2010; J acob & Ludwig, 2008; Yeh, 2016). One model of urban 
school reform aimed at leveling achievement disparities in the public school systems that has 
garmered attention (both positive and negative) is the No Excuses model. The No Excuses model is 
driven by the philosophy that there should be no excuses (e.g., poverty) for low academic 
performance. This model has found its footing in charter schools—publicly funded but privately 
managed schools exempt from the restrictions of the laws and regulations affecting non-charter 
public schools. Due to the autonomy afforded to the management of charter schools, educators 
have the freedom to implement the tenets of the No Excuses model without the restrictions placed 
on non-charter public schools. 


Along with the expansion of charter schools, overall, No Excuses charter schools have been 
increasing in number since their recognition in the 1990s. No Excuses charter schools primarily 
draw from low-income and minority communities with the aim of minimizing racial and income- 
based achievement gaps by improving the academic performance of their students. Although a 
definitive definition of the No Excuses charter school model does not exist, most definitions share a 
number of key characteristics—most notably: (a) high academic expectations, (b) rigid and 
consistent discipline, (c) extended instructional time, (d) intensive teacher training, and (e) 
parental involvement (Dobbie & Fryer, 2013; Whitman, 2008; Carter, 2000; Golann, 2015). 


OBJ ECTIVES 


This systematic review and meta-analysis aims to examine the available evidence on the impacts of 
No Excuses charter schools on students’ math and literacy achievement, relative to students 
enrolled in traditional public schools. The focal research questions are as follows: 


e Does the No Excuses charter school model enhance students’ math or literacy achievement 
more so than traditional public schools? 

e What is the magnitude and variability of the effects? 

e What are the limitations to the evidence? 
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SEARCH METHODS 


We conducted a comprehensive systematic literature search to identify all eligible studies, 
regardless of publication status. For the purposes of this review, the term “studies” is used to refer 
to separate reports and “samples” to refer to independent samples of students within (and over 
multiple, in some cases) studies. The search was conducted using electronic databases, internet 
search engines, citations in previous meta-analyses and literature reviews, citations in research 
reports screened for eligibility, conference listings, hand searches of relevant journals, and 
correspondence with experts in the field. We screened redactions and erratum for potentially 
eligible studies. We used date-of-publication restrictions limiting our search years to 1990 or later 
as No Excuses charter schools did not exist until the mid- 1990s. 


Selection criteria 


Published or unpublished studies using randomized controlled trials (RCTs) or quasi-experimental 
design (QEDs) to assess the effects of No Excuses charter schools on math and literacy 
achievement of elementary and secondary students were included in this review. To be eligible for 
inclusion in this review, studies must have been conducted or reported between J anuary, 1990 and 
June, 2016. The date range reflects the time period during which No Excuses charter schools have 
been present in the U.S. School system. In addition, studies must have reported or controlled for 
baseline and comparison data, and comparisons groups must have comprised students attending 
traditional public schools. Studies determined not to reflect charter schools using a No Excuses 
model, or comparing No Excuses charter schools to other charter schools were excluded from this 
review. 


DATA COLLECTION AND ANALYSIS 


A team of reviewers assessed each title and abstract for preliminary eligibility, which was 
confirmed during full-text screening. All studies that met the eligibility criteria in the citation and 
abstract, and full-text screening were coded using a detailed coding instrument developed by the 
authors. Data extracted from primary studies included information on methodology, design, 
intervention and comparison group characteristics, math or literacy outcome details and 
characteristics, etc. Coders met to review the coding agreement and any discrepancies were 
discussed and resolved by consensus. This review contains a description of the intervention, a 
summary of the authors’ findings and conclusions, and a narrative synthesis of the identified 
literature. 


We report standardized mean differences with Hedges’ g. Separate meta-analyses were performed 
to synthesize the studies assessing the effects of No Excuses charter schools on math outcomes and 
to synthesize the studies assessing the effects of No Excuses charter schools on literacy outcomes. 
Separate meta-analyses were used to estimate the cumulative effects after one, two, three, and four 
years of attendance at No Excuses charter schools. Meta-analyses separately report the estimated 
effects from RCTs and QEDs. We calculated Q, t2, and I? to assess heterogeneity. 
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Moderator analyses were planned to explain between-study heterogeneity in effects for several 
school-level moderator variables, including percent male, percent minority, percent free/ reduced 
lunch, percent of students provided special education services, school size, average class size, and 
years that the charter school has been active. However, few studies reported these characteristics at 
the school level; further, several variables (such as percent male, percent minority, percent 

free/ reduced lunch, and percent special education services) did not have sufficient variability 
between studies to serve as useful moderators for effect sizes. As such, no planned moderators were 
consistently significant predictors of math or literacy achievement gains. 


RESULTS 


The initial search of both academic and grey literature identified a total of 4,554 citations. After 
duplicates and removal of studies prior to 1990, 2,841 citations remained. Titles and abstracts were 
screened for initial relevance and 2,386 of the remaining reports were excluded for irrelevance or 
ineligibility. Full-texts of the remaining 455 studies were retrieved (no unretrievable reports) and 
screened for eligibility—18 of which passed full-text screening for inclusion in the primary review 
resulting in 73 independent samples over four years. Analysis of these samples examined the 
effects of No Excuses charter schools on the math and literacy achievement of 70,121 elementary 
and secondary students. Five of the included studies were RCTs (Abdulkadiroglu et al. 2011; 
Angrist et al. 2010; Curto & Fryer, 2014; Hastings, Nekilson, & Zimmerman, 2012; Tuttle et al., 
2015) and the remaining 13 used quasi-experimental designs. The weight given to RCTs in the 
overall estimates (which pool the QED and RCT effects) of math and literacy effects were 
approximately 8% for year 1, 7% for year 2, 6% for year 3, and 5% for year 5. Studies included 
nongovernmental reports (9), journal articles (5), dissertations (2), working papers (1), and papers 
presented at conferences (1). All studies evaluated results using comparison groups comprised of 
students attending traditional public schools. 


The effect of No Excuses charter schools on math and literacy outcomes was found to be significant 
in this meta-analysis. Similar to previous findings, a substantially smaller but significant effect on 
literacy was found. 


Math results 


The year 1 math analyses included 73 separate samples. The overall mean effect size for math 
outcomes was positive and significant (g =0.202, 95% CI [0.148; 0.257]). The mean effect sizes 
were significant for QEDs (g =0.207, 95% CI [0.148; 0.266]) and RCTs (g =0.149, 95% CI [0.015; 
0.283]) and were not significantly different from each other. There was significant heterogeneity in 
the estimates of the effect size (t2 =0.0430). Given the significant heterogeneity and potential bias 
for non-equivalent groups, the mean effect size should be interpreted with caution (I? =89.4, Q = 
676.58). At follow-up years, the mean effect on math outcomes remained significant: Year 2 (n = 
48), g =0.293, p <0.001; Year 3 (n =33), g =0.434, p <.001; Year 4 (n =22), g =0.299, p< 
0.001. The estimated mean effects after adjustment for potential publication bias using trim-and- 
fill for all four years were also significant: Year 1 (0 trimmed, g = 0.207, p <0.001), Year 2 (4 
trimmed, g =0.250, p <0.001), Year 3 (6 trimmed, g =0.331, p <0.001), Year 4 (0 trimmed; g = 
0.299, p <0.001). 
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Literacy results 


The year 1 literacy analyses also included 73 separate samples. The overall mean effect size for 
literacy outcomes was positive and significant (g =0.069, 95% CI [0.034; 0.104]). The mean effect 
sizes were significant for QEDs (g =0.061, 95% CI [0.025; 0.097]) and RCTs (g =0.164, 95% CI 
[0.019; 0.309]) and were not significantly different from each other. Heterogeneity was minimal (12 
= 70.6, T? =0.0121, Q =244.77). At follow-up years, the effect on literacy outcomes remained 
significant and attenuated from the math results: Year 2 (n =48), g =0.120, p <0.001; Year 3 (n = 
33), g =0.212, p <.001; Year 4 (n =22), g =0.178, p <0.001). The Egger’s test indicated potential 
publication bias for the Year 1 literacy results (p =.047). After adjusting the mean effect using 
trim-and-fill, the Year 1 mean effect was still significant (7 trimmed, g = 0.041, p =0.028), as were 
all follow-up years: Year 2 (6 trimmed, g = 0.091, p <0.001), Year 3 (3 trimmed, g = 0.168, p < 
0.001), Year 4 (1 trimmed, g = 0.178, p <0.001). 


AUTHORS’ CONCLUSIONS 


The present review identified available evidence regarding the effectiveness of No Excuses charter 
schools on math and literacy achievement. This review found multiple studies providing tentative 
support for the No Excuses charter school model as an effective intervention for improving 
students’ math and literacy achievement, with limitations. Further, this intervention was found to 
improve students’ math achievement more significantly that literacy. 


There were several significant limitations to this review. First and of utmost importance, studies 
were inconsistent in reporting (or failed to report) important sample characteristics (e.g., percent 
of students receiving special education services, number of students exiting or expelled after 
enrollment, etc), severely limiting interpretation of the equality of comparison groups and factors 
impacting differences in outcomes. Second, follow-up years suggest a sustained significant effect 
on both math and literacy achievement, however these results must be interpreted with caution as 
the number of samples decreased substantially at follow-up years with one study (Tuttle, Teh, 
Nichols-Barrer, Gill, & Gleason, 2010), accounting for the majority of the samples. Third, studies 
included in this review did not report data indicating the degree to which each characteristic of No 
Excuses charter schools was implemented. This is a significant limitation as it prohibits the 
interpretation of the presence, absence, or dosage of the individual characteristics of No Excuses 
charter schools. Fourth, only five of the 18 included studies (approximately 5.75% of the 
independent samples) employed random assignment, limiting interpretation of the equality of the 
comparison groups due to the small number in independent samples using random assignment. 
Fifth, a large number of the included studies examine the effects of the schools of a specific charter 
Management organization: Knowledge is Power Program (KIPP). Specifically, approximately 57% 
of the weight in Year 1 effect size estimates came from KIPP samples. At year 2, 3, and 4 follow-ups, 
more that 57% of the weight in the effect size estimates are from KIPP samples. Though 
informative, the results of this review cannot be considered generalizable to all No Excuses charter 
schools. Of note, the results of this review reflect one unique model of charter schools and must not 
be generalized to all charter schools. Finally, most studies measured effects with only one, annual 
standardized achievement measure. 
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1 Background 


11 THE PROBLEM, CONDITION OR ISSUE 


111 Persistent academic achievement gaps 


In response to the school accountability movement and the move toward market-oriented 
education policies in recent decades, many schools and districts have felt increased pressure to find 
effective ways to boost student achievement. However, many districts—largely those serving low- 
income black and hispanic students in urban settings—have still struggled to make sufficient 
academic gains (Barton & Coley, 2010; Jacob & Ludwig, 2008; Yeh, 2016). The prevalence of this 
achievement gap is one of the most critical current issues in U.S. education and contributes 
significantly to racial and economic inequality (Duncan & Magnuson, 2013). 


Racial and economic disparities develop in large part because children from low-income or 
minority (i.e., Black and Hispanic) households disproportionately manifest poor academic 
achievement in school. According to the Achievement Gap Initiative at Harvard University, 
disparities in family resources predict between one-half and two-thirds of the total racial 
achievement gap (Achievement Gap Initiative, 2013). Poverty rates in 2014 were 26.2% for Black 
and 23.6% for Hispanics, compared to 10.1% for non-Hispanic whites and 12% for Asians 
(DeNavas- Walt & Proctor [U.S. Census Bureau], 2014). Academic achievement gaps are both the 
causes and the consequences of the disparity of resources. 


According to the National Assessment of Education Progress Long Term Trend Assessment 

(NAEP; U.S. Department of Education, 2014), the academic performance of both 9- and 13-year- 
olds has improved across all groups since the 1970s, and gaps between groups have narrowed. 
However, the gaps between Black and Hispanic 17 year-olds and their white peers, which narrowed 
during the 1970s and 1980s, have remained stable since 1990 without explanation, and the income- 
based achievement gap has widened significantly over the last three decades (Reardon, 2013). 
Despite periods of progress, racial and income-based achievement gaps remain substantial (Barton 
&Coley, 2010; J acob & Ludwig, 2008). They begin early in life, and, given the rapidly growing 
population among the lowest- achieving groups (U.S. Census Project, 2010), these gaps will only 
become more critical in the future. 
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12 THE INTERVENTION: NO EXCUSES CHARTER SCHOOLS 


12.1 Charter schools: an overview 


The current school choice model intends to provide families with alternatives to publicly provided 
neighborhood schools by offering options such as school vouchers, scholarship tax credits, open 
enrollment laws, and charter schools. Beginning with the first charter written in Minnesota in 1991, 
charter schools have emerged as a educational reform concept that has attracted significant 
attention—and scrutiny—from education leader and reformers. A charter school is a publicly 
funded, semi-autonomous school that typically operates under a legislative contract with a 
"sponsor" (e.g., State, district, other entity). This contract—or charter—details school management, 
student expectations, and achievement measures while providing exemption from laws and 
regulations affecting non-charter public schools. In return for this flexibility and autonomy, the 
charter school must meet the accountability standards outlined in its charter. A school's charter is 
reviewed periodically by the entity that granted it and can be revoked if guidelines on curriculum 
and management are not followed or if the accountability standards are not met. Laws governing 
charter schools vary among states, and often differ on important factors, such as who is allowed to 
authorize charter schools, accountability for student outcomes, and whether to require certification 
for charter school teachers (Thomsen, 2016). 


According to the U.S. Department of Education (2016), between 2000 and 2014, the enrollment of 
American public school students in charter schools as a percent of total enrollment in public 
schools grew from 0.7% (0.3 million) to 5.4% (2.7 million). Along with this growth in enrollment, 
the number of charter schools as a percent of total public schools increased from 1.7% to 6.9%. This 
data suggests that charter schools are poised for continued growth and expansion, making it crucial 
for educators, policymakers, researchers, and other stakeholders to develop a deeper 
understanding of their efficacy on student outcomes. Given the autonomy granted to charter 
schools in their operation and management, charters nationwide vary significantly among states, 
among districts, and among individual schools. This wide variation among charters makes it 
difficult to evaluate overall impact or draw broad inferences from nationwide data. This review 
aims to address this issue by exploring one subtype of charter school—No Excuses charter 
schools—allowing for comparison between charter schools that employ similar models of 
management, instruction, and school culture. 


12.2 No Excuses charter schools 


A growing number of charter schools in urban communities across the United States have 
employed a No Excuses philosophy, which focuses intensely on improving the math and literacy 
achievement of students who come primarily from low-income and minority backgrounds. The 
formal discussion of schools defined as "No Excuses" began as a national effort organized by The 
Heritage Foundation to mobilize public pressure on behalf of better education for the poor (Carter, 
1999). Samuel Casey Carter, a Bradley Fellow at the Heritage Foundation, brought attention to the 
initiative in his book No Excuses: Lessons from 21 High-Performing, High-Poverty Schools 
(2000). The philosophy was further heralded by Thernstrom and Thernstrom in their book No 
Excuses: Closing the Achievement Gap In Learning (2003). The No Excuses philosophy is driven 
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by the philosophy that there is no excuse for the academic failure of schools serving low-income 
children and is aimed to embolden educators to develop school-based interventions and schooling 
models aimed at improving academic outcomes for this population (Carter, 1999). This model has 
found its footing in charter schools—publicly funded but privately managed schools exempt from 
the restrictions of the laws and regulations affecting non-charter public schools. Due to the 
autonomy afforded to the management of charter schools, educators have the freedom to 
implement the tenets of the No Excuses model without the restrictions placed on non-charter 
public schools. Since their introduction around the turn of the millennium, aspects of the No 
Excuses model have spread widely through the charter school landscape; in some states, charter 
schools that adhere somewhat or fully with No Excuses principals make up two-thirds of the urban 
charter schools (Angrist, Pathak, & Walters, 2013) 


Several charter networks such as the Knowledge is Power Program (KIPP) and Uncommon Schools 
have received national attention by espousing a “no excuses” philosophy, and many lesser-known 
charter schools have adopted a similar model, such as the North Star Academy of Newark, New 
Jersey, and the Amistad Academy in New Haven, Achievement First, and Promise Academy 
(Armor, 2004; Golann, 2015). The adoption of the No Excuses model primarily in charter school 
settings is likely due to the operational autonomy afforded to charter schools, allowing charter 
Management organizations the freedom to implement innovative models within their schools. 


Although a consensus definition of the No Excuses charter school model has not been adopted, 
most definitions appear to center around common key characteristics. For the purposes of the 
present review, we will define a No Excuses charter school using these five common key 
characteristics: (a) high academic expectations, (b) rigid and consistent discipline, (c) extended 
instructional time, (d) intensive teacher training, and (e) increased parental involvement (Dobbie 
& Fryer, 2013; Thernstrom & Thernstrom, 2003; Whitman, 2008; Carter, 2000; Golann, 2015). 
For the purposes of this review, a charter school will be defined as No Excuses if it employs each of 
these five key characteristics. However, schools may adopt and employ these characteristics to 
varying degrees, along with additional policies that may distinguish them from other schools. 
Additional characteristics outlined by Carter (2000) may include autonomy of administrators, use 
of measurable goals, employment of master teachers, rigorous and regular testing, and 
requirement of mastery for grade promotion, among others. 


12.3 Characteristics of No Excuses charter schools 


12.3.1 High academic expectations 


A common thread in the discussion of the reasons underlying the achievement gap is expectations, 
as both teacher and parental expectations can influence their students' academic achievement (Fan 
&Chen, 2001; Jussim, Madon, & Chatman, 1994; Rosenthal &J acobson, 1968). Education 
stakeholders have speculated that expectations are reduced for low-income and minority students 
(Tenenbaum & Ruck, 2007) due to teachers’ implicit biases against ethnic minority students (Van 
den Bergh, Denessen, Hornstra, Voeten, & Holland, 2010) and beliefs that minority students' poor 
performance stems from innate characteristics rather than situational factors (J ackson, 2002). 
Specifically, in No Excuses charter schools, teachers are asked to increase expectations of students 
by benchmarking their students’ achievement against that of students in highly successful 
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classrooms across the country (Farr, 2010). As teachers' expectations have their greatest effects in 
classrooms in which high-achieving and low-achieving students are treated very differently 
(Kulkinski & Weinstein, 2001), the No Excuses charter school model attempts to address the 
potential for lower achievement by ensuring that expectations are universally high for all students. 
Further, Fan and Chen (2001) found a strong relationship between parental expectations and their 
childrens’ academic achievement. Thus, the high academic expectations set forth by the No 
Excuses charter schools may influence parents’ expectations, further enhancing achievement 
outcomes. Within this model, high academic expectations are often met through intensive 
remediation of core subjects (e.g., intensive tutoring). 


12.3.2 Rigid and consistent discipline 


Rigid and consistent discipline appears to be a common and primary component of many No 
Excuses charter schools. Rigid disciplinary codes are often reflected in mandated dress codes and 
schoolwide, highly structured, consistently enforced conduct expectations; setting standards for 
how students walk in the hallways, demonstrate attentiveness in class, and organize their school 
work (Goodman, 2013; Lake et al., 2012; Whitman, 2008). 


Research on the effects of school uniforms have not been compelling (Yeung, 2009). Several 
studies have either found school uniforms to be negatively correlated with academic performance 
(Brunsma & Rockquemore, 1998; Brunsma, 2004) or have found little to no significant association 
between uniforms and performance (e.g., Barton, Coley, & Wenglinsky, 1998; Gentile & Imberman, 
2012; Sanchez, 2012). 


In terms of conduct expectations, Lake and colleagues (2012) conducted a national study of charter 
management organization effectiveness and found a positive association between student behavior 
policies and math and reading achievement in four No Excuses charter school management 
organizations. Further, consistency of enforcement of behavioral expectations may play an 
important role in the effectiveness of No Excuses charter schools’ disciplinary strategies as 
inconsistency of enforcement has been associated with behavior problems (Feehan, McGee, 
Stanton, & Silva, 2011; Stormshak, Bierman, McMahon, & Lengua, 2000). The rigid disciplinary 
approach used in the model continues to be controversial. Proponents suggest that the disciplinary 
approach is necessary to promote self-discipline and personal responsibility in an effort to teach 
students traditional, middle-class values (Whitman, 2008). Opponents of the disciplinary 
approach are concerned that the sole focus is compliance—undermining students’ self confidence 
and identities, disallowing students the ability to learn to this for themselves, the rules are too 
numerous, and students are disciplined through a culture of shame (Ellison, 2012; Goodman, 
2013; Lack, 2009). 


12.3.3 Extended instructional time 


No Excuses charter schools (and many other school models) make an effort to treat instructional 
time as an educational resource. However, measuring different aspects of the impact of extended 
instructional time has proved challenging for researchers. Findings related to its impact on student 
achievement appear to agree that there is little or no relationship between allocated time and 
student achievement (Aronson, Zimmerman, & Carlos, 1999; Checkoway et al., 2012). However, 
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when used for academic learning, extended instructional time has been shown to exert positive 
impacts on student achievement (Aronson et al., 1999; Farbman, 2012; Farbman & Kaplan, 2005). 


12.3.4 Intensive teacher training 


Teachers employed at No Excuses charter schools often participate in intensive pre-service training 
before the beginning of a new school year and ongoing professional development throughout the 
year in the form of intensive teacher coaching and monitoring. New teachers are observed by 
master teachers, receive feedback on their performance, and submit lesson plans for review (Lake 
et al., 2012). Despite several decades of research, the relationship between teacher effectiveness 
and teacher training has not been conclusively demonstrated for traditional school models. 
However, Lake and colleagues (2012) found intensive teacher coaching to be an effective strategy 
for four high-performing No Excuses charter school management organizations. 


12.3.5 Parental involvement 


The parental accountability implemented in No Excuses charter schools may play a key role in 
affecting student achievement. Many studies have found that students with involved parents were 
more likely to earn higher grades and test scores, no matter what their income or background 
(Fehrman, Keith, & Reimers, 1997; J eynes, 2012; Steinberg, Lamborn, Dornbusch, & Darling, 
1992). Several studies found that families of all income and education levels, and from all ethnic 
and cultural groups, are engaged in supporting their children’s learning at home. White, middle- 
class families, however, tend to be more involved at school (Henderson & Mapp, 2002). Low- 
income and minority families, however, often face barriers to sufficient involvement in their 
children’s schooling such as monetary constraints, time constraints, transportation needs, 
language and other communication differences, and cultural beliefs about the role of family in 
children’s schooling (Boethel, 2003). Critical questions have been raised regarding how parental 
involvement is conceptualized, especially within low-income and minority communities (Barton, 
Drake, Perez, St. Louis, George, 2000). Social and cultural contexts must be taken into 
consideration when defining parental involvement. 


The common parental accountability component of the No Excuses charter school model attempts 
to minimize barriers and cultivate commitment from parents to reinforce school actions with 
strategies such as written contracts explaining the school’s expectations regarding (a) parental 
responsibilities, (b) academic standards, and (c) conduct and discipline, for example; requiring 
homework assignments to be checked and signed by parents; parent orientation meetings, and 
asking that the parent check and sign their child’s homework before returning it to the teacher. 
However, further research is required to provide evidence that parental involvement requirements, 
such as those employed in a No Excuses charter school, are sufficient for overcoming the barriers 
noted. 


12.4 Use of best practices 


Research examining best practices from both charter school and traditional literatures finds that 
successful schools exhibit similar characteristics regardless of their status as public, private, or 

charter (Fryer & Leavitt, 2004; Maas & Lake, 2015). The success achieved by No Excuses charter 
schools may be the result of implementation of a key combination of best practices. For example, 
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Maas and Lake compared current knowledge about the characteristics of effective charter schools 
to effective school research. They found several best practices found in the education literature that 
appear to closely align with some of the key characteristics of No Excuses charter schools: (1) high 
expectations, (2) increased time, and (3) a focus on student learning. This demonstrated success of 
best practices across settings (i.e., charter and non-charter) suggests that stakeholders focus on the 
implementation of these practices in schools independent of the schools' status as a charter or non- 
charter. 


13 WHY IT IS IMPORTANT TO CONDUCT THIS REVIEW 


Taking into account the accumulating literature suggesting the potential benefits of No Excuses 
charter schools, there is a need to comprehensively synthesize the full evidence base of 
methodologically rigorous studies that examine impacts of No Excuses charter schools on math 
and literacy achievement. A systematic review that evaluates the efficacy of the model will advance 
the evidence base and provide researchers, educators, and policymakers with a means by which to 
make informed decisions about the use of the model in the field. 


As students’ poor academic achievement is associated with lower earning potential (Miller, 1998), 
greater youth violence (Hawkins, Herrenkohl, Farrington, Brewer, Catalono, & Harachi, 1998; 
Kingery, Pruitt, Heuberger, & Brizzolara, 1996), increased drug use (Cox, Zhang, J ohnson, & 
Bender, 2007; Mensch & Kandel, 1988), and delinquency (Maguin & Leober, 1996), identifying 
means of decreasing the achievement gap and improving all students' educational performance is 
an important priority for educational research. 


Given the expanded recent interest in school choice and charter schools—and specifically in No 
Excuses charter schools—as an instrument of education reform for low-income and minority 
students, now more than ever, there is need for critically evaluating information about their 
performance. A growing body of controversial evidence suggests that No Excuses charter schools 
may represent a promising strategy for addressing the racial and income-based achievement gaps 
(Abdulkadiroglu et al., 2009; Angrist, et al., 2011; Dobbie & Fryer, 2013), but also suggests that the 
small-scale success experienced by schools implementing this model may be the result of factors 
that do not permit the model to be scalable (e.g., high teacher attrition, disproportionate number of 
highly-dedicated teachers) (Yeh, 2016). Thus, the present review is necessary to determine if this 
model of schooling precipitates these purported effects in order to ensure that students are justly 
served by its use. 


To our knowledge, one prior review was conducted to synthesize the effects of No Excuses charter 
schools on math and literacy achievement (Cheng, Hitt, Kisada, & Mills, 2017). The present review 
intends to expand upon the previous study primarily in four ways: (1) the present review represents 
amore inclusive, comprehensive, and systematic literature search and includes unpublished 
studies; (2) more relevant inclusion criteria were used to improve the tenability of the results; and 
(3) our results included theses and dissertations. 
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2 Objectives 


The systematic review and meta-analysis aims to examine the available evidence on the impacts of 
No Excuses charter schools on students’ math and literacy achievement relative to students 
enrolled in traditional public schools. The focal research questions are: 


e Does the No Excuses charter school model enhance students’ math or literacy achievement 
more so than traditional public schools? 

e What is the magnitude and variability of the effects? 

e What are the limitations to the evidence? 


Secondarily, should the included studies provide us with sufficient relevant data to explore 
differential effects for No Excuses charter schools with different characteristics (i.e., potential 
moderators such as percent minority, percent male, school and class size, etc.), we will explore 
their presence and magnitude. 
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3 Methods 


3.1 CRITERIA FOR SELECTING STUDIES 


Eligible studies met all criteria set forth below for inclusion in this systematic review and meta- 
analysis. 


3.11 Types of studies 


Included studies used experimental RCTs or controlled QEDs. For QED studies to be eligible, 
researchers must have compared No Excuses charter school students to demographically similar 
students who attended nearby traditional public schools. Methods that can be used to establish the 
demographic similarity of groups included: individual or school-level matching on at least one 
measure of race, ethnicity, socioeconomic status, or achievement; regression models that use 
student fixed effects to adjust for student race, ethnicity, or socioeconomic status; or instrumental 
variables analyses that adjust an instrument for whether students enrolled in a charter school. 


Eligible QED studies were required to report baseline achievement for the intervention and 
comparison conditions. Baseline equivalence, defined as less than one-fourth of a standard 
deviation difference between the means in the intervention and comparison at baseline 
achievement, was also measured and reported by study overall. If baseline achievement data were 
not reported, then the authors were required to use baseline achievement to match subjects (such 
as with propensity score methods) or to statistically adjust for difference in baseline achievement 
in their analyses (e.g., to adjust group means or as a covariate in a regression model). 


3.12 Types of participants 


This review specifically examined students enrolled in elementary and secondary No Excuses 
charter schools. The student populations attending No Excuses charter schools are predominantly 
low-income and/ or minority (Frankenberg, Siegel- Hawley, Wang, & Orfield, 2012; Tuttle et al., 
2010). 


3.13 Types of interventions 


This review examined charter schools that implemented a No Excuses model, both in the U.S. and 
abroad. To determine if a charter school used a No Excuses model, it was evaluated for the 
presence of five key characteristics discussed above [i.e., (a) high academic expectations, (b) rigid 
and consistent discipline, (c) extended instructional time, (d) intensive teacher training, and (e) 
increased parental involvement]. All five key characteristics were required to be present for a 
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school to be identified as a No Excuses charter school. These characteristics must be mentioned 
explicitly as being endorsed by the charter school, although leniency was granted for the degree of 
operationalization (e.g., what is “rigid and consistent” discipline and “intensive” teacher training). 
This method of defining a No Excuses charter school included schools that unknowingly conformed 
to the No Excuses model without including charter schools that only employ part of the No Excuses 
model. Because our interest is in No Excuses charter schools, studies that compare achievement in 
public schools to charter schools, without adequate information to separate No Excuses charter 
schools from charter schools employing other models, were not eligible. 


3.14 Types of outcome measures 


Eligible studies measured at least one standardized student-level math or literacy achievement 
outcome. All meta-analyses were conducted separately for the math and literacy outcomes 
(described in more detail in the synthesis section below). 


Math and literacy outcomes standardized to national, state, or another representative samples 
were used. These can be defined as standardized assessments on which students demonstrate 
grade-level appropriate knowledge of math or literacy domains. For example, math or literacy 
subtests of the NAEP, SAT, ACT, PISA were all acceptable standardized measures. Raw scores, 
standardized scores, and percentile scores were all acceptable outcomes. All unstandardized 
outcomes, such as curriculum-based measurements, grade point averages, or teacher-reported. 
grades, were excluded from the review to minimize potential bias from such unstandardized 
assessments. 


3.L5 Additional criteria 


Studies that only combined math and literacy outcomes into a unified achievement measure were 
excluded from the review. Consistent with previous findings on No Excuses charter schools (Cheng 
et al., 2017), we expected that math achievement gains might not be identical to literacy 
achievement gains. 


Studies must have been conducted after 1990. This is the year the first charter school was founded, 
and thus we do not expect any studies to have been conducted prior to this date. 


Eligible studies may have been conducted in any country. However, the cultural factors informing 
the development and implementation of No Excuses charter schools largely limited the eligible 
studies to the heterogeneous population of the United States. 


To be included in the quantitative synthesis, studies were required to report statistics necessary for 
calculating effect sizes. If multiple forms of an eligible outcome were reported (such as means and 
standard deviations as well as pass rates), we used the maximally informative form. If both 
adjusted effect sizes (accounting for differences between the groups on baseline achievement, 
demographics, or both) and unadjusted effect sizes were reported, we used the adjusted effect 
sizes. 
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3.L6 Duration of follow-up 


We conducted separate meta-analyses for each yearly follow-up to estimate cumulative effects of 
No Excuses charter school attendance. Included studies reported sufficient information for four 
yearly follow-ups: Year 1, Year 2, Year 3, and Year 4. 


3.L7 Types of settings 


Studies conducted in natural school settings were eligible for inclusion in this review. Studies 
conducted in non-school settings were ineligible due to external validity concerns including the 
potential limitation in providing generalizability of findings to the target population. 


3.18 Types of comparison groups 


Comparisons groups must have been comprised of students attending traditional public schools. 
Studies comparing No Excuses charter schools to other charter schools are excluded from this 
review. Charter school policies and practices (e.g., funding rules, operational requirements, 
accountability) vary widely across states and even within districts, while traditional public schools 
are held to similar rules and regulations. A comparison of No Excuses charter schools and charter 
schools overall would limit interpretation of the results. 


3.L9 Timeframe of studies 


Studies must have been conducted or reported between J anuary, 1990 and J une, 2016. The date 
range reflects the time period during which No Excuses charter schools have been present in the 
U.S. School system. 


3.110 Study locale 


Eligible studies may have been conducted in any country. However, the cultural factors informing 
the development and implementation of No Excuses charter schools limited the eligible studies to 
the heterogeneous population of the United States. 


3.2 SEARCH METHODS FOR IDENTIFICATION OF STUDIES 


3.2.1 Electronic searches 


We used the ProQuest database host to search 15 electronic databases: 


PsycINFO, PsycARTICLES, ProQuest Social Sciences Premium Collection (includes ERIC), 
ProQuest Dissertations & Theses Global, ProQuest Sociology, ProQuest Social Science J ournals, 
ProQuest Science J ournals, ProQuest Religion, ProQuest Psychology J ournals, ProQuest Political 
Science, ProQuest Health and Medical Complete, ProQuest Family Health, ProQuest Education 
Journals, ProQuest Criminal J ustice, PILOTS, and ebrary e-books. 


The search strategy used to search ProQuest is outlined in Table 3.1. Beyond including charter and 
charter schools in the search, we included specific brands of charter schools in the United States, 
such as KIPP, Rocketship Education, and Great Hearts. To identify international studies, we used 
the search terms free, paternalistic, autonomous, foundation, and academies. Through extensive 
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scoping searches, we found these to be the most common terms used for charter schools 
internationally. We suspected the term free school would draw a particularly diverse swath of 
irrelevant studies, including literature related to smoke-free schools, free school meals, and the 
Free School Movement. Thus, we excluded those terms from the title search. 


To ensure comprehensive inclusion of relevant studies, we limited search terms about design and 
outcomes to only the broadest terms. While we hoped to target well-described studies with terms 
such as lottery, propensity, or experiment*, we also included terms such as match*, compar*, and 
control* to target studies that do not describe their methodology in great detail. To search eligible 
standardized achievement outcomes, we searched standard*, achiev*, and assess* as well as 
effect*, affect, impact* and outcome*. 


Table 3.1. Search Strategy in ProQuest 


Search Description 


Limited to title and ti,ab("charter school" OR "charter schools" OR charter* OR "KIPP" OR 

abstract "Achievement First" OR "Uncommon schools" OR "Match Education" 
OR "Friendship public" OR "Breakthrough Schools" OR "Brighter 
Choice" OR "IDEA public" OR "YES prep" OR "Alliance of College 
Ready" OR "Alliance College-Ready" OR "Rocketship Education" OR 
"Summit Public" OR "Green Dot" OR "Great Hearts" OR "STRIVE Prep” 
OR "Strive Preparatory" OR "Aspire Public" OR "Bright Star" OR 
"Leadership Public" OR "Success Academies" OR "Summit public" OR 
"Synergy Academies" OR ((foundation OR free OR paternal* OR 
autonom*) PRE/1 school*) OR (("academy" OR "academies") AND 
("United Kingdom" OR "England" OR "British" OR "Britain"))) AND 
ti,ab(“lottery" OR compar* OR random* OR control* OR propensity OR 
match* OR experiment*) AND ti,ab(achiev* OR assess* OR math* OR 
liter* OR read* OR effect* OR outcome* OR standard* OR affect* OR 
impact*) 


Excluded from title = NOT ti(lunch* OR meal* OR breakfast* OR drug* OR smok* OR bully* 
OR psychiatr*) 


Additional Date: After 1 January 1990 
delimiters 


3.2.2 Searching other resources 


In addition to the electronic search of the above databases, we reviewed the references lists of all 
retrieved documents. The websites of relevant professional organization such as Mathematica, and 
American Institutes for Research (AIR) were searched for potentially eligible research reports. 
Finally, key investigators who are known to be active in the field and experts identified during the 
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review process were contacted with a request to share any published, unpublished, and ongoing 
research relevant to the review. 


3.3 DATA COLLECTION AND ANALYSIS 


3.3.1 Data extraction and management 


Two reviewers independently extracted data from at least 30% of the eligible studies. Prior to 
coding, the authors participated in comprehensive training involving review of coding rules and 
areas of ambiguity in the coding manual. Periodic reliability assessments to protect against coder 
drift were conducted during the initial stages of coding. Inter- rater reliability checks were 
conducted prior to coding the final set of eligible studies. Data extraction forms for reliability, 
coding, and screening were developed. If a study was found ineligible at the full-text coding stage, 
the reasons for ineligibility were documented and reported. Disagreements in coding were resolved 
through discussion and consensus; any disagreements about coding items were resolved through 
discussion with the third review author. Before synthesis of results, effect sizes were assessed for 
numerical accuracy to ensure extracted effect sizes match the direction of effects reported. 


3.3.L1 Potential moderators 


To ensure robustness of findings, several study and design characteristics were pre-identified and 
extracted for inclusion in moderator analyses. These moderator analyses were only estimated if at 
least five studies contributed to a given subset. Data was extracted for the following potential 
moderator variables: (1) study source (e.g., journal, independent report), (2) achievement measure 
type (i.e., state or national), (3) percent of sample that was minority status, (4) percent of sample 
that was male, (5) percent of sample receiving free-and-reduced lunch (FRL), (6) percent of sample 
receiving special education services (SPED), ('7) class size (i.e., student-to-teacher ratio), (8) school 
size, (9) number of years charter has been active, (10) charter school network status (i.e., if the 
charter school was part of a charter management organization), and (11) attrition. 


Study design (RCT vs. QED) and study source (e.g., journal, dissertation, unpublished study) were 
included to determine if there are significant differences between randomized and non-randomized 
findings and to determine if unpublished reports—commonly sponsored by charter organizations— 
have significantly different findings from peer-reviewed published articles or dissertations. The 
types of reported achievement measures were also included, separating state-standardized from 
nationally standardized tests, to determine if findings differ for measures which vary widely in 
difficulty from state to state. 


The school-level characteristics of No Excuses charter schools may help to identify school qualities 
that promote the effectiveness of the No Excuses model, as well as provide important information 
about the scalability of the No Excuses model. For instance, student gender was also included as 
moderator variables as it has been theorized that the high-expectations and strict discipline within 
No Excuses charter schools may help male students' performance more than female students’. 
Thus, we may expect different math or literacy gains from No Excuses charter schools with 
different percentages of males. Futher, based on prior work showing that No Excuses charter 
schools (and charter school overall) enroll a lower percentage of students with disabilities 
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(Lacireno-Paquet, Holyoke, Moser, & Henig, 2002; Miron, Urschel, & Saxton, 2011; Nichols- Barrer 
et al., 2016), special education status was included to determine if effects differ in relation to the 
proportion of students receiving special education services. 


3.3.12 Details of study coding categories 


All studies that met the eligibility criteria in the citation and abstract, and full-text screening were 
coded using a detailed coding instrument developed by the authors. Data extracted from primary 
studies included information on methodology, design, intervention and comparison group 
characteristics, math or literacy outcome details and characteristics, etc. See Appendix I for coding 
manual. 


3.3.13 Criteria for the determination of independent findings 


We took several steps to ensure that all effect sizes used in our meta-analysis represent statistically 
independent observations. First, we collated multiple studies that report on a single sample; three 
studies were found that report on the same samples. Second, if a study reported multiple eligible 
math or literacy measures we averaged the measures into a single outcome within that outcome 
domain using the procedure outlined above (Borenstein et al., 2009 p.230). However, no study 
reported multiple eligible math or literacy outcomes in a given year. Finally, there is a risk that 
large-scale studies (such as Tuttle et al., 2010) that included multiple independent samples may 
overlap samples studies in smaller, single-school studies. To avoid over-weighting these samples 
(which represent a small fraction of the included samples), meta-analyses used robust standard 
errors for the large-scale studies. Coders also made every effort to re-identify the de-identified 
schools that might be over-represented and to confirm that the years and grades reported in each 
study did not overlap. 


3.3.2 Statistical procedures and conventions 


3.3.2.1 Measures of treatment effect 


Most studies reported continuous measures of math or literacy outcomes, so we used the small- 
sample corrected standardized mean difference effect size metric (Hedges’ g) to estimate the mean 
difference in outcomes by intervention and comparison conditions. If an eligible study reported 
binary outcomes (e.g., pass-fail rates for an achievement test) by condition, we computed log odds 
ratios; then converted these log odds ratios to Hedges’ g effects (using procedures outlined in 
Borenstein et al., 2009, p. 47) and synthesized these effect sizes in a single meta-analysis for each 
outcome type. Several studies reported either standardized or unstandardized regression 
coefficients for the effect of No Excuses attendance. For these effect sizes, we converted used the 
Campbell Collaboration effect size calculator 

(http:// www.campbellcollaboration.org/ escalc/ html/ EffectSizeCalculator-SMD-main.php) to 
convert these effect sizes to Cohen’s d, and then converted to Hedge’s g (see Table 4.4). 


3.3.2.2 Unit of analysis issues 


For all included studies, the unit of assignment and the unit of analysis were at the level of the 
individual student. 
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3.3.2.3 Dealing with missing data 


For studies missing sufficient data for calculation of effects sizes, we solicited authors for the 
needed information or made reasonable estimates from the reported data, if possible. If authors 
did not respond and reasonable estimates could not be made, these studies were excluded from the 
synthesis, but provide helpful information in the systematic review. 


3.3.2.4 Assessment of heterogeneity 


We calculated Q, 12, and I? to assess heterogeneity. The Q statistic is distributed as a chi-square 
with k- 1 degrees of freedom (k =the number of effect sizes). 


3.3.2.5 Data synthesis 


Given the presumed heterogeneity in included studies, we used random-effects meta-analysis 
models to synthesize findings across studies. All analyses were conducted separately for the two 
outcomes: math and literacy. All analyses were performed using meta-analysis commands that run 
in the Stata statistical environment (StataCorp, 2011). 


3.3.2.6 Moderator and subgroup analysis 


For each subset of studies (i.e., outcome type by follow-up year), we conducted meta-regression 
models to explore the potential effect of each of the moderator variables (see section 3.3.1.1 for 
discussion of included moderators). We examined the magnitude and statistical significance of the 
slope coefficients from the meta-regression models to assess moderator effects. For moderator 
analyses, we pooled effect size estimates for QEDs and RCTs for each year follow-up, and only 
conducted meta-regressions for each moderator if at least four samples reported moderator 
information. While meta-regressions with multiple moderators as predictors would have provided 
an even richer exploration of the school- and study-level moderators, multicollinearity between the 
moderators, as well as unreported moderator data and lack of moderator variability, prevented 
extensive moderator analyses. Results for the moderator analyses are reported in Section 4.4. 


3.3.2.7 Sensitivity analysis 


We defined outliers as effect sizes three standard deviations away from the median. In Year 1, we 
flagged one sample from Woodworth et al. (2008) as an outlier on both math and literacy. In Year 
3, we flagged one sample from Mac Iver & Farley-Ripple (2007) on math and one sample from 
Tuttle et al. (2010) on literacy as outliers, respectively. No outliers were detected for Years 2 and 4. 
Meta-analyses were re-ran with the outliers removed and with the outliers Winzorized to three 
standard deviations above the mean. In both cases, the results were nearly identical to those 
conducted without changing the outliers. As such, we report the results from the analyses with the 
outliers included unchanged. 


3.3.2.8 Detecting and adjusting for publication bias 


We conducted Egger’s tests to detect publication bias. Because Egger’s test can be underpowered in 
small samples, we used trim-and-fill analyses even when Egger’s test was non-significant to adjust 
for potential publication bias. 
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4 Results 


4.1 RESULTS OF THE SEARCH 


4.11 Included studies 


Figure 4.1 shows the study identification flow diagram reflecting the study identification process 
for all identified research reports for the current review. Electronic searches of bibliographic 
databases and searches of other sources (e.g., citation search, etc.) identified a total of 4,554 
citations with 44 additional records identified through other sources and search methods. After 
duplicates and removal of studies prior to 1990, 2,841 citations remained. Titles and abstracts were 
screened for initial relevance and 2,386 of the remaining reports were excluded for irrelevance or 
ineligibility. Full-texts of the remaining 455 studies were retrieved (no unretrievable reports) and 
screened for eligibility—18 of which passed full-text screening for inclusion in the primary review. 


4.12 Excluded studies 


During the full-text screening of the study selection process, 438 studies were excluded. The 
majority of these exclusions were due to the involved schools not employing a No Excuses model— 
an inclusionary criteria which could not often be determined during screening of the title and 
abstract. During full-text screening, if the study did not explicitly state that the school in the 
samples employed a No Excuses model, we examined the school descriptions reported in the study 
for the presence of the five characteristics of the No Excuses model. In the event that these 
characteristics were not explicitly reported, we explored the mission statements for or related 
information about the specific schools or schools included in the study to determine if the school 
either explicitly used the No Excuses model or reported implementation of the five characteristics 
of the No Excuses model identified for the purposes of this study. The remaining studies were 
excluded due to (a) not meeting criteria for study design, such as matching without using baseline 
achievement, or (b) reporting of outcomes irrelevant to our review (e.g., course grades). We further 
identified five potentially eligible reports that were conducted in the United Kingdom. However, we 
were unable to determine the employment of the No Excuses model in these studies and they were 
therefore excluded. 
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Figure 4.1. PRISMA flowchart illustrating study identification process 
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4.2 DESCRIPTION OF THE STUDIES 


The results of our search are comprised of nongovernmental reports (9), journal articles (5), 
dissertations (2), working papers (1), and papers presented at conferences (1). The included studies 
employed RCTs (n =5; ~5.75% of independent samples) and QEDs (n = 13; ~94.25% of 
independent samples). All 18 studies were conducted in the United States and used state-level 
standardized tests to measure students’ achievement, as opposed to national standardized test 
(e.g., SAT, ACT). See Table 4.1 for a summary of the included studies. 
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Table 4.1. Summary of included studies 


Separate Grade Charter 


Follow- 


Charter Students 


Siudy Design PUBIE auOnaLYPe Samples Levels Schools up Years in First Year peas 
Abdulkadiroglu et al. (2011) RCT Journal 2 M,H 8 41 2,634 No 
Angrist, et al. (2010) RCT Journal 1 M 1 2 856 Yes 
Crosby (2015) QED Dissertation 1 H 2 1-5 179 Yes 
Curto & Fryer (2014) RCT Journal 1 M,H 1 1 129 No 
Dobbie, Fryer, & Fryer (2011) QED Journal 2 E,M 1 1 1,098 No 
Fryer (2014) QED Journal 3 E,M,H 20 1-3* 4,824 No 
Gallagher & Ross (2005) * QED Report 3 M 1 1-2* 117 Yes 
Hastings, Nekilson, & Zimmerman (2012) RCT Working Paper 1 E,M,H 8 1 237 No 
Herman et al. (2011) QED Report 3 8 1-3* 1,119 No 
Mac Iver & Farley-Ripple (2007) QED Report 4 M 1 1-4* 355 Yes 
McDonald, Ross, Abney, & Zoblotsky (2008) * QED Conference Paper 10 M 1 1-4* 246 Yes 
Rose (2013) QED Dissertation 1 M 1 1-4 90 Yes 
Ross, McDonald, & McSparrin-Gallagher (2004) * QED Report 1 M 1 1 49 Yes 
Teh, McCullough, & Gill (2010) QED Report 1 M 5 1-3 300 Yes 
Tuttle et al., (2010) QED Report 22 M 22 1-4* 11,228 Yes 
Tuttle et al., (2015) RCT/QED Report 3 E,M,H 38 1-3* 2,289 Yes 
Woodworth, et al. (2008) QED Report 6 M 3 1 279 Yes 
Young et al., (2009) QED Report 19 E,M 23 1-2* 2,670 No 


Note. E=elementary, M=middle, H=high school. Studies report on some of the same samples. 
“Indicates not all samples report on all follow-up years. 
tT Studies report on some of the same samples. Information on the fullest sample characteristics available from each report were coded. 
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4.3 RISK OF BIAS IN INCLUDED STUDIES 


4.3.1 Potential sample bias 


The basic research designs used in eligible studies were random assignment using enrollment 
lotteries at oversubscribed schools and comparing lottery winners with lottery losers (RCTs) or 
matching students for baseline equivalence on achievement and demographic characteristics 
(QEDs). Studies using random assignment via lottery may be largely free of the self-selection bias 
that often plagues studies of this type. However, only 5 of the 18 studies used random assignment, 
representing approximately 8% of the independent samples in Year 1. The remaining 13 QED 
studies demonstrated baseline equivalence on achievement, but may still be affected by self- 
selection bias (or other types of sample bias) despite the researchers efforts to equate matches. 


Thus, the potential for sample bias remains a concern in the interpretation of the results for the 
majority of the studies, as unreported differences likely exist between students electing to enroll in 
a No Excuses charter school and students attending traditional schools. Sample biases may result 
primarily from differences in attrition (Miron et al., 2011) or differences in population of students 
with exceptional needs such as English language learners and students with learning disabilities 
(Lacireno-Paquet et al., 2002; Miron et al., 2011; Nichols- Barrer et al., 2016). To attempt to 
mitigate some of the potential sample bias, intent-to-treat (ITT) effects were coded, if available, 
rather than the more liberal treatment-on-treated (TOT) effects. See Section 5.4.1 for further 
discussion of sample biases. 


4.3.2 Analysis of potential publication bias 


Reporting (publication) bias stems from failing to detect unpublished studies. Underrepresentation 
of unpublished studies, which are more likely to have non-significant effects, can substantially bias 
effect size estimates (Borenstein et al., 2009). The majority of studies included in this review are 
nongovernmental reports conducted by independent research organizations. While the large 
proportion of unpublished studies in this review may suggest the absence of publication bias, the 
presence of a majority proportion of reports on large, multi-site studies may suggest the absence of 
smaller studies with null effects or the exclusion of samples within the multi-site studies in which a 
non-significant effect were found. We examined potential publication bias using Egger’s 
regression- based assessment of asymmetry of funnel plots (see Figures 4.2 through 4.9; Egger, 
Smith, Schneider, & Minder, 1997). While many samples with small standard errors (indicating, in 
general, large sample sizes) show large effect sizes, there does not appear to be any bias in these 
samples toward larger effects or asymmetry in the plots. Because the funnel plots indicated no 
difference in publication bias between RCTs and QEDs, we conducted the Egger’s tests on the 
combined results for math and literacy effects separately. 


4.3.2.1 Publication bias of math results 


The Egger’s test did not find sufficient evidence of bias for math results for any year (Year 1, p = 

0.979, Year 2, p =0.325; Year 3, p =0.283; Year 4, p =0.851). These results are not confirmatory 
of the absence of publication bias. The Egger’s test can be underpowered in small samples such as 
those analyzed in Years 2, 3, and 4. Further, the results from Years 3 and 4 are from samples from 
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only a few multi-site studies. However, they do suggest that its presence is unlikely and allow for 
increased confidence that few small sample studies with null or negative findings were omitted 
from the current analysis. Trim-and-fill analyses were conducted for each year. The estimated 
mean effects for each year after trimming-and-filling were consistent with the untrimmed analyses: 
Year 1(0 trimmed, g = 0.207, p <0.001), Year 2 (4 trimmed, g =0.250, p <0.001), Year 3 (6 
trimmed, g = 0.331, p <0.001), Year 4 (0 trimmed; g =0.299, p <0.001). 


4.3.2.2 Publication bias of literacy results 


The Egger’s test found evidence of potential publication bias for the effect of the intervention on 
literacy in Year 1 (Year 1, p = .047). While Years 2, 3 and 4 do not show evidence of publication bias 
(Year 2, p =.144, Year 3, p =.283; Year 4, p =.851), again, we do not take this as confirmatory 
evidence of lack of publication bias. Using the trim-and-fill method to adjust for potential 
publication bias, estimated mean effects were attenuated but still significant and consistent with 
the untrimmed results: Year 1 (7 trimmed, g =0.041, p =0.028), Year 2 (6 trimmed, g =0.091, p < 
0.001), Year 3 (3 trimmed, g =0.168, p <0.001), Year 4 (1 trimmed, g = 0.178, p <0.001). 


Figure 4.2. Funnel plot (math outcomes - Year 1) 


Funnel plot with pseudo 95% confidence limits 
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Figure 4.3. Funnel plot (math outcomes - Year 2) 
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Figure 4.4. Funnel plot (math outcomes - Year 3) 
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Figure 4.5. Funnel plot (math outcomes - Year 4) 


Funnel plot with pseudo 95% confidence limits 
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Figure 4.6. Funnel plot (literacy outcomes - Year 1) 
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Figure 4.7. Funnel plot (literacy outcomes - Year 2) 


Funnel plot with pseudo 95% confidence limits 
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Figure 4.8. Funnel plot (literacy outcomes - Year 3) 
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Figure 4.9. Funnel plot (literacy outcomes - Year 4) 


Funnel plot with pseudo 95% confidence limits 
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4.3.3 Risk of bias in the randomized control trials 


For the five studies employing a RCT, we investigated reporting bias (via selective reporting), and 
other potential bias sources. We found no evidence of reporting bias as all five RCTs investigated 
standardized achievement measures and reported both math and literacy achievement. 


4.4 SYNTHESIS OF RESULTS 


Overall Hedges’ g mean effect size measures by year and achievement outcome are reported Table 
4.2. Figure 10.1.1 through 10.2.4 present the forest plots for the math and literacy outcomes by year 
and by assignment type. The forest plots are organized by effect size, and include standard error 
bars for each sample. Sample effect sizes, confidence intervals, and weights are expressed in the 
plots as well. However, only five of the 18 eligible studies were RCTs. As such, the number of 
independent samples implementing RCTs was small for Year 1 and was even smaller for Years 2, 3, 
and 4. Table 4.3 provides further information about the effect sizes gathered from each study, 
including types of effect sizes reported (mean differences, pass rate, regression coefficients, etc.) 
and covariate adjustment of effect sizes. 
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Table 4.2. Hedges’ g mean effect sizes and heterogeneity statistics by outcome type and year 


Outcome N Hedges’ g p 95% Cl Q P r 
Math 
Year1 73 0.202 0.000 0.15;0.26 676.58 89.4 0.043 
Year2 48 0.293 0.000 0.22;0.36 574.16 91.8 0.048 
Year3 33 0.434 0.000 0.34;0.53 461.54 93.1 0.063 
Year4 22 0.299 0.000 0.19;0.41 325.85 93.6 0.058 
Literacy 
Year1 73 0.069 0.000 0.03;0.10 244.77 70.6 0.012 
Year2 48 0.120 0.000 0.08;0.17 263.91 85.3 0.020 
Year3 33 0.212 0.000 0.14;0.29 313.98 89.8 0.040 
Year4 22 0.179 0.000 0.13;0.24 107.80 80.5 0.016 


4.4.1 Mean effects on math outcomes 


The Year 1 math results indicated that the overall mean effect size for math achievement gains was 
positive and significant (g =0.202, 95% CI [0.148; 0.257]). After one year of No Excuses charter 
school attendance, students outperformed comparable peers who attended public schools by 0.202 
standard deviations. The estimates of the effect size were heterogeneous, with a variance 
component of r2 equal to 0.043 that is significantly different from zero. Given the significant 
heterogeneity and potential bias for non-equivalent groups, the mean effect size should be 
interpreted with caution (I? =89.4, Q =676.58). These results indicate that attending a No Excuses 
charter school is associated with about one-fifth of a standard deviation improvement over the 
course of a year. At follow-up years, the mean effect on math outcomes remained significant: Year 
2, g =0.293, p <0.001; Year 3, g = 0.434, p <.001; Year 4, g =0.299, p <0.001. Interestingly, we 
observed a general increase in math achievement improvement from Year 1 to Year 3, at which 
point the math achievement gains stabilized or returned to lower gains. 


4.4.2 Mean effects on literacy outcomes 


The Year 1 literacy results indicated that the overall mean effect size for literacy achievement gains 
was positive and significant (g =0.069, 95% CI [0.034; 0.104]), although much smaller than 
effects associated with math achievement. After one year of No Excuses charter school attendance, 
students outperformed comparable peers who attended public schools by 0.069 standard 
deviations, or about one-fourteenth of a standard deviation. Again, heterogeneity was significant 
and due to true between- study differences, but was quite small (I2 = 70.6, Q =244.77, r2 =0.0121). 
These results indicate that attending a No Excuses charter school is associated with a small, but 
statistically significant, increase in literacy achievement. At follow-up years, the effect on literacy 
outcomes remained significant and attenuated from the math results: Year 2, g =0.120, p <0.001; 
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Year 3, g =0.212, p <.001; Year 4 (n = 22), g =0.178, p <0.001). Much like the math results, 


though, we observed that students’ achievement gains increased year-on-year until Year 4, at which 


point literacy achievement gains stabilized or decreased. 


Table 4.3. Types of effect sizes and covariate adjustment by study 


Study 


ES reported 


Covariate Adjustments 


Abdulkadiroglu et al. (2011) 


Regression coefficient 


Baseline achievement, 
demographics 


Angrist, et al. (2010) 


Crosby (2015) 


Regression coefficient 


Means and SDs 


Baseline achievement, 
demographics 


None+ 


Curto & Fryer (2014) 


Regression coefficient 


Baseline achievement, 
demographics, year of test 


Dobbie, Fryer, & Fryer 
(2011) 


Regression coefficient 


Baseline achievement, 
demographics, year of test 


Fryer (2014) 


Regression coefficient 


Baseline achievement, 
demographics, year of test 


Gallagher & Ross (2005) t 


Means and SDs 


Baseline achievement 


Hastings, Nekilson, & 
Zimmerman (2012) 


Regression coefficient 


None? 


Herman et al. (2011) 


Mean Difference* 


Baseline achievement 


Mac Iver & Farley-Ripple 
(2007) 


Regression coefficient* 


Baseline achievement, 
demographics 


McDonald, Ross, Abney, & 
Zoblotsky (2008) t 


Means and SDs 


Baseline achievement 


Rose (2013) 


Percent Passed Rates 


None+ 


Ross, McDonald, & 
McSparrin-Gallagher (2004) 
* 


Means and SDs 


Baseline achievement 


Teh, McCullough, & Gill 
(2010) 


Regression coefficient 


Baseline achievement, 
demographics, year of test 


Tuttle et al., (2010) 


Regression coefficient 


Baseline achievement, 
demographics, year of test 


Tuttle et al., (2015) 


Regression coefficient 


Baseline achievement, 
demographics, year of test 


Woodworth, et al. (2008) 


Means and SDs 


None+ 


Young et al., (2009) 


Regression coefficient 


Baseline achievement, 
demographics 


* SDs not reported, but were estimated from the reported p-values. 
tT SDs for achievement not reported, but estimated from a state report on the achievement measure. Authors 


have been contacted for exact SDs. 
*Demonstrate equivalence between the groups on baseline achievement (<0.25 SD difference between 


group means). 
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4.5 MODERATOR ANALYSES 


Plans for moderator analyses on several school- and study-level characteristics are outlined in 
Section 3.3.2.6. Many studies failed to report moderator data for independent samples, such as 
school size, class and class size, preventing analyses for these moderators. Further, several 
moderators (state versus national standardized test, and if the charter school was part of a charter 
Management organization) showed insufficient variability among the included studies to conduct 
moderator analyses. However, we were able to conduct meta-regressions to explain variability in 
math and literacy gains with several moderators: number of year charter has been active (Years 
Active), percent of sample that was minority status (% minority), percent of sample that was male 
(% male), percent of sample receiving free/ reduced lunch (% FRL), and percent of sample receiving 
special education services (% SPED). 


Table 4.4 reports the zero-order meta-regression coefficients for moderators by year. None of our 
planned moderators consistently predicted heterogeneity in math or literacy effects. This is likely 
due to lack of variability in the moderator variables. For example, the majority of the included 
samples had high percentages of minority students and students receiving free or reduced lunch— 
this is the student demographic No Excuses schools were designed to support. Further, in Years 2, 
3, and 4, many of the independent samples were reported from a few large, multi-site studies, 
which further limited variability. As such, we recommend caution in interpretation of the non- 


significant moderator analyses. 


Table 4.4. Zero-order meta-regression coefficients for moderators 


Math 
Year 1 
Year 2 
Year 3 
Year 4 

Literacy 
Year 1 
Year 2 
Year 3 


Year 4 


Years Active 


-0.034 (.04) 
-0.267 (.13) 
-0.079 (.21) 


N/A 


0.011 (.01) 
-0.044 (.09) 
-0.095 (.13) 


N/A 


% Minority 


-0.005 (.01) 
0.026 (.03) 
0.320 (.24) 


0.024 (.06) 


-.004 (.01) 
0.014 (.02) 
0.184 (.13) 


0.013 (.02) 


% Male 


-0.013 (.01) 
-0.017 (.02) 
-0.023 (.04) 


0.056 (.06) 


-0.001 (.01) 
-0.020 (.01) 
-0.002 (.02) 


0.014 (.02) 


% FRL 


-0.005 (.01) 
0.007 (.02) 
0.010 (.04) 


0.001 (.05) 


-0.006 (.003) 
0.009 (.01) 
0.013 (.02) 


0.005 (.01) 


% SPED 


-0.013 (.02) 
-0.045 (.04) 
0.131 (.31) 


N/A 


-.004 (.01) 
-0.016 (.02) 
0.030 (.18) 


N/A 


Note. “Years Active” refers to the number of years the charter school in each sample has been active. % 

Minority, Male, Free/Reduced Lunch (FRL), and Special Education (SPED) indicate school- and/or sample- 
level percentages. Standard errors are shown in parentheses, bolded coefficients are significant at a = .05. 
Samples sizes vary by year and moderator due to unreported data. N/A indicates either no variability in the 
moderator or less than four independent samples for the given meta-regression. 
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After data collection, we observed that several of the multi-site studies reported solely on KIPP 
schools. As such, we conducted post-hoc, separate meta-analyses for included KIPP versus Non- 
KIPP No Excuses charter schools for math and literacy effects for Year 1 (Figures 4.10 and 4.11). 
Approximately 57% of the overall math and literacy effect sizes were due to KIPP samples. Effect 
estimates from KIPP schools were stronger (math, g = 0.32, 95% CI [0.23; 0.40]; literacy, g =0.09, 
95% CI [0.05; 0.14]) than for non-KIPP schools (math, g =0.06, 95% CI [0.00; 0.12]; literacy, g = 
0.04, 95% CI [-0.01; 0.09]). This result may either indicate publication bias favoring KIPP versus 
Non-KIPP No Excuses charter schools or may represent true differences in achievement gains due 
to implementation of the No Excuses model in KIPP versus Non-KIPP No Excuses charter schools. 
See Figures 9.1.5 and 9.2.5 for Forest plots for KIPP versus non-KIPP math and literacy outcomes. 
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5 Discussion 


5.1 SUMMARY OF MAIN RESULTS 


The present review evaluated eighteen studies examining the effects of No Excuses charter schools 
on math and literacy achievement. The findings of the review point to positive and significant 
effects on both math and literacy, with math gains larger than literacy gains. Our findings indicate 
that one year of attending a No Excuses charter school is associated with approximately one-fifth of 
a standard deviation improvement in math performance and one-fourteenth of a standard 
deviation in literacy performance. However, follow-up analyses indicated that these benefits 
accrued for three years, which point the achievement gains stabilized or returned to lower gains. 
The present finding of higher estimated effects for math outcomes has been demonstrated in the 
prior No Excuses charter school review (Cheng et al., 2017); that this pattern of findings is 
observed for the No Excuses charter school model lends confidence to the results. The potential 
presence of sample bias limits our confidence in these results. 


53.2 OVERALL COMPLETENESS AND APPLICABILITY OF EVIDENCE 


Our screening of 2,841 citations from across academic and grey literature yielded 18 eligible 
studies. The comprehensiveness of our search and ability to obtain all identified articles for review 
and screening makes us confident that we identified all available literature. However, this also 
suggests that there are significant limitations to the availability of rigorous, high-quality evidence 
for this intervention. 


Interpretation of the available evidence is limited by sample bias and missing data and thus does 
not allow us to draw strong, generalizable conclusions regarding the effects of the intervention. 


5.3 QUALITY OF EVIDENCE 


Studies meeting eligibility criteria for this review were required to meet stringent methodological 
standards (i.e., RCTs and QEDs). 


5.4 LIMITATIONS 


5.4.1 Sample biases 


Studies were inconsistent in reporting (or failed to report) several important sample characteristics 
(e.g., school size, percent of students receiving special education services, attrition). The student 
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populations of No Excuses charter schools (and charter schools in general) have been shown to 
differ in important ways from their traditional public school counterparts. The lack of this 
important information limits the interpretability of the results of this review. 


First, students with more motivation to be academically successful or students with more parental 
involvement (factors difficult to observe and measure) may elect to enter charter lotteries at higher 
rates than the average student attending a traditional public school—traits that would likely affect 
academic achievement regardless of school model. 


Second, charter schools as a whole appear to use expulsion and suspension as a mechanism of 
discipline enforcement at a higher rate than non-charters (Losen & Keith, 2016). These practices 
may result in lower-performing or less-compliant students exiting the schools, leaving behind a 
population of students more compliant to the practices of the No Excuses model—again, the traits 
of the compliant students would likely affect academic achievement regardless of school model. In 
the first comprehensive report of charter school discipline, Losen, Keith, Hodson, and Martinez. 
(2016) examined data from the 2010-2011 school year for 89,750 public non- charter schools and 
more than 5,250 charter schools. They found the national charter school suspension rate to be 16% 
higher than the non-charter school suspension rate. Further, data from the office of Civil Rights 
suggest that more than 17% of all secondary-level charter schools reportedly suspended zero 
students, leading Losen and colleagues to suspect that a high number of secondary charter schools 
might not have reported their data. 


Third, students with exceptional needs such as English language learners, students with 
disabilities, or students with behaviour problems are often a smaller proportion of students 
attending all types of charter schools than traditional public schools (Lacireno- Paquet et al., 2002; 
Miron et al., 2011; Nichols- Barrer et al., 2016). Charter schools have been shown to initially enroll a 
lower percentage of students with disabilities than traditional public schools (Scott, 2012). For 
example, during 2008-2009, KIPP schools (charter schools specifically employing the No Excuses 
model) enrolled a lower percentage of students with disabilities (5.9%) and English language 
learners (11.5%) than did their local school districts (12.1% and 19.2%, respectively) (Miron et al., 
2011). This important factor that is likely to inflate the academic performance of a charter school. 
Further, the high expectations, rigid disciplinary policies, and higher rates of expulsion and 
suspension at No Excuses charter schools may reduce the population of students with exceptional 
needs through attrition, leaving these students to fall behind and rejoin their peers in higher 
concentrations at traditional public schools. 


Finally, the available research on the rates of attrition of No Excuses charter schools indicates 
higher rates than traditional public school counterparts (e.g., Casey, 2016; Miron et al., 2011; 
Nichols-Barrer et al., 2016). Although the studies in this review include RCTs, it is important to 
consider that lottery winners and losers who appear similar at the time the lotteries were held, 
subsequent attrition may lead to important differences in the follow-up sample if the attrition 
process is nonrandom. The analysis conducted by Miron et al. on KIPP schools during 2008-2009 
revealed that, on average, approximately 15% of students exit the KIPP grade cohorts each year. 
These three analyses also each observed that fewer students are replaced in the later years of 
middle school. In one example, Casey reports the graduating class of Harlem Success Academy I 
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had 32 students in eighth grade during 2013-20 14—less than half of the 73 students who started in 
the cohort eight years prior. Further, Nichols-Barrer and colleagues (2016) observed that KIPP 
schools tended to replace exiting students with higher-achieving students. These practices are 
likely to result in significant differential attrition, specifically in the higher grades. 


5.4.2 Differences in teacher quality 


A recent analysis conducted by Yeh (2017) suggests that the observed effects of No Excuses charter 
schools on achievement are likely due to a "hoarding of a disproportionate share of the nation’s 
limited pool of highly-dedicated teachers, rather than gains that could be sustained when the 
programs are scaled up and implemented nationwide" (pp. 98-99). Although we extracted all 
available teacher data, included studies did not report measures of teacher quality and dedication, 
disallowing the ability of this review to explore the this hypothesis. 


5.4.3 Disproportionate number of samples examining KIPP 


Further, a large number of the included studies (10 of the 18 included studies and approximately 
58% of the independent samples) examine the effects of KIPP schools, specifically. The prevalence 
of KIPP schools in this body of evidence is expected, as KIPP schools—although not the first to use 
it—were the first to become widely known for implementing a No Excuses model. However, in 
addition to implementing the policies common to other No Excuses charter schools, KIPP schools 
presumably also have features unique to KIPP schools. Though informative, the results of this 
review cannot be considered generalizable to all No Excuses charter schools. 


5.4.4 Studies limited to U.S. schools 


During the literature search, extensive efforts were made to thoroughly search the grey literature. 
This process is designed to collect as many potentially relevant as possible and to mitigate against 
any potential threat of reporting (publication) bias. Our comprehensive and indiscriminate 
searches of international literature provided studies examining charter school performance; 
however, upon full-text screening, we were unable to decisively determine whether the potential 
international studies were assessing charter schools using the No Excuses model due to limited 
reporting of charter school model details. Thus, this review may have missed relevant extant 
international literature and results cannot be generalized to other populations. 


5.4.5 Additional limitations 


Follow-up years suggest a sustained significant effect on both math and literacy achievement, 
however these results must be interpreted with caution as the number of samples decreased 
substantially at follow-up years, with one study (Tuttle et al., 2010), accounting for the majority of 
the independent samples. 


Studies included in this review did not report data indicating the degree to which each 
characteristic of No Excuses charter schools was implemented. This is a significant limitation as it 
prohibits the interpretation of the presence, absence, or dosage of the individual characteristics of 
No Excuses charter schools. 
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Only five of the 18 included studies (approximately 5.75% of the independent samples) employed 
random assignment, limiting interpretation of the equality of the comparison groups due to the 
small number in independent samples using random assignment. 


A large number of the included studies (10 of the included studies; approximately 58% of the 
independent samples) examine the effects of the schools of a specific charter management 
organization: Knowledge is Power Program (KIPP). ‘Though informative, the results of this review 
cannot be considered generalizable to all No Excuses charter schools. Finally, 


Additional limitations include the following: (a) the majority of studies included in this review 
employed QEDs, thus increasing the threat of potential bias associated with QEDs, (b) included 
studies were limited to the U.S., reducing generalizability of results to other countries which may 
be implementing a No Excuses model within its charter schools, (c) most studies measured effects 
with only one, annual standardized achievement measure, and (d) the lack of a clear definition of a 
No Excuses charter school led us to develop a comprehensive set of key characteristics with which 
to determine a charter school’s model type, introducing the potential for mislabeling a charter 
school’s model. 


5.3 AGREEMENTS AND DISAGREEMENTS WITH OTHER STUDIES OR 
REVIEWS 


To our knowledge, only one prior review of No Excuses charter schools has been conducted using 
systematic review procedures as a working paper for the Education Reform (EDRE) Department at 
the University of Arkansas (Cheng et al., 2017). Findings are in agreement, suggesting that No 
Excuses charter schools may be more effective than traditional public schools in increasing 
achievement in math and literacy. 
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6 Authors’ conclusions 


The present review synthesized multiple rigorous studies with numerous independent samples and 
provides initial evidence that the No Excuses charter school model may be effective as an 
intervention for math and literacy outcomes, as measured by standardized achievement measures. 
Data indicates that these benefits accrue for three years, which point the achievement gains 
stabilized or returned to lower gains. 


Strong potential for sample bias and the presence of further limitations restricts interpretability of 
the results. Further, most studies measured effects with only one, annual standardized 
achievement measure and thus the evidence suggesting sustained intervention effects is more 
limited. Additional research examining school-level characteristics beyond those available for this 
review is necessary for thorough investigation of the causes of achievement gains for No Excuses 
charter schools. 


6.1 IMPLICATIONS FOR PRACTICE AND POLICY 


These findings appear to have the greatest implications for those seeking to expand the No Excuses 
model to additional schools and to apply tenets of the model to existing educational environments. 
Because these results tentatively indicate positive effects associated with the No Excuses model, 
some policymakers and educators may wish to explore the possibility of incorporating aspects of 
the No Excuses model into the current educational landscape and may wish to lobby for the 
expansion of the model. Thus, we urge that policymakers and practitioners take these results with 
the caution, especially in consideration of the publication bias and heterogeneity present in these 
findings. 


6.2 IMPLICATIONS FOR RESEARCH 


The results of this review have substantial implications for future research and should be of 
interest to researchers, educators, education reformers, administrators, policymakers, and 
other stakeholders invested in closing gaps in achievement and improving academic outcomes 
for students. 


First, additional research is needed to determine the scalability of the No Excuses model. Prior 
studies exploring scalability have returned conflicting results. Yeh (2017), conducted a reanalysis of 
the Knowledge is Power Program (KIPP) and the Harlem Children’s Zone Promise Academies and. 
found that gains would fall to zero upon full scale-up. In contrast, Tuttle and colleagues (2015) 
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examined an effort to scale up the Knowledge is Power Program (KIPP) with results suggesting 
gains in student academic achievement can be maintained with program expansion. 


Second, studies using more rigorous controls for self-selection and sample bias are necessary to 
determine the true effect of the No Excuses model on achievement. Third, while these findings 
suggest that students attending No Excuses charter schools tend to outperform peers attending 
traditional public schools, the individual key characteristics of No Excuses charter schools critical 
to the No Excuses model are not yet understood. Future research may involve component analyses 
to determine if certain key characteristics have a higher impact on outcomes than others. Finally, 
future research in these areas may be able to inform the field regarding instructional practices, 
interventions, and other strategies the may help to reduce the achievement gaps. 
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9 Figures 


9.1 FOREST PLOTS FOR MATH OUTCOMES 
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9.1.1 Math outcomes - Year 1 


es (98% C4 wor 


rong ee a 


Young et a 


Vang ea 


McDereid et 3 (2008) 


o-} gM -o---------ly pt Lipman Tee PET | I 


younger a3i 
Tue etal 165 
Yangad 107 
young eta jaa 
Young et 4 14 
POUND ef a 06 (0.34, 0.23) 123 
Tume etal (2010 4 (-0.24,0 14) 151 
Tums ebat (0TH 1-021, 010) 15) 
Veang ef a (20087) r 3.098) 199 
Tun 2070) 5.01 $0,11, 6:03) 1 

Young et a 7 1a) i 

angers J 4) ' 

Time etal 10 (-0.16, 0 18) eu 
Young ot a 001 C08, 097; 102 


Tum etl 160 
Young era 59 
Henan eta jag 
Ture or uh (2091 ve 
Youryy et a, (2008) 154 
MeDored ef at (2008) 4 


ore eta (20 


Tune etal (2 


11 aan 1236 

ow 13) 00, 150 

014 Om) (a6 

4 2, oso 

0.15 0.06 al 

Mae liner & faneysripany (200 0 13 eu 10, Oath ie 
ave (ats) 19 4 17, Cagy 2 
Tums ef at (2010 0600 2) 158 
cungeta (2005) 018 126 
049 131 

2.14 (0.08, aay 10 

0.19 (005.034) 102 

(9 (2.027) (77 

0 1 86 

0 7) 

1a 


Tube et a) (2090 


Tumse etal (2010 Ox 2 170 

Woodworth ef al (2008 6.35 (0.42 063) 190 
Tut etal (7000 % (0-23.G03) 1 

Rows, McDoruht, & MicSpanreGudiagner (7 # 0 Ue, 75) ow 

Crosby (2015) 0.3 (0 19.084) 152 
Hermarret a (2014 O37 (0.44, Ge) i 

0.43 (027,004) 157 

m1 (aa 019,010 eo] 


Nopaworts etal 


MeDorwidt et at (N08) 


Tume et D 


Fanay Rinp 
Wooeneveaty el, (ZH $4 (0.24, 8) 08 
Tum etal 12010) 0.56 (040.972) 97 
Tut et al GI @ 47,074) (Ra 
Tube et al 062 (051 672) 4 
D7 (0.41, 1.00) 119 
0.73 W459, Ofa) 162 
nv austin mn 
2.993) wn 

oor 11 (79,951 a 

Suntotal (I-aquared = 69 6%, 0 =0 21 (016,02 og21 


Haslnes, Nelson, & Limmernnan (2012) v.03 a7 

Tume etal DoIS (oc 126) 

Fryer (2014) 0.14 (0.07, 620) (78 
uta & Fryer( 014 0 104.0 12 


AULT 


Shiota (l.squareo = 7 


6) (00.0 


ayara pun Favors charter 


55 The Campbell Collaboration | www.campbellcollaboration.org 


9.1.2 Math outcomes - Year 2 
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Citation ES (95% Cl) Welaht 

' 
QED t 
Rose (2013) { -0:24 (-0.57_ 0.08) 165 
Young et al. (2009) { +620 (-0,32, -0.07) 238 
Young etal. (2002) tl -0.15 (-0.34, 0,03) 219 
Young et al, (2009) tl -0.03 (+0, 19,,0.14) 225 
Tuttle et al (2010) t 0.02 (-0.17_ 0.13) 230 
Young et al, (2009) 0.00 (-0.32, 0,33) 1.67 
Turtle ef al. (2010) 0.00 (-0.19, 0.19) 2.18 
Young et al. (2003) 0.03 (-0.22, 0.28) 1.34 
Tuttle et al. (2010) 0.03 (-0.16, 0.22) 218 
Young et al, (2002) 0.06 (-0.18, 0,30) 1,97 
Tuttle et al. (2010) 0.07 (-0.06, 0.20) 237 
Young et al. (2009) 0.09 {-0.24, 0.41) 1.64 
Teh, McCullough, & Gill (2070) 0.13 (-0.03, 0,29) 227 
Tuttle et al, (2070) 0.13 (0.03, 0,23) 245 
Fryer (2014) 0.13 (0.06, 0,20) 2.53 
Tuttle et al. (2070) 0.17 (0.05, 0,29) 2. 
Tuttle et al (2010) 0.17 (0.06, 0,28) 2.44 
Fryer (2014) 0.17 (0.13. 0.21) 2.57 
Young et al. (2009) 0.26 (-0.10,,0.61) 1.53 
Tuttle et al (2010) 0.27 (0.15, 0.39) 2.40 
Tuttle et al. (2015) 0.28 (0.17, 0,38) 244 
Tuttle et al. (2010) 0.28 (0.16. 0.41) 2.39 
McDonald et al, (2068) 0.30 {-0.22, 0.82) 1.05 
Hernan etal. (2011) 030 (0.05. 0.55) 193 
McDonald et al, (2008) 0.31 (0,68, 1,30) 0.41 
Tuttle et al. (2010) 0.31 (0.20, 0.43) 2.41 
Tuttle et al. (2070) 0.33 (0.20, 0.47) 2.35 
Tuttle et al_ (2010) 033 (023, 0.44) 2.45 
Crosby (2015) 0.39(0.20, 0,57) 220 
Gallagher & Ross (2005) 0.40 (-0.04, 0.84) 4.27 
Tuttle et al. (2070) 0.40(0.23, 0,56) 2.26 
Tuttle ef al. (2010) 0.40 (0.29, 0,50) 2.44 
McDonald et al. (2008) 0.45 (-0.86, 1.76) 0.25 
Tuttle et al. (2010) 0.47 (0.31, 0,63) 227 
Tuttle et al. (2010) 0.54 (0.43. 0.65) 2.43 
Tuttle et al. (2010) 0.59(0.46, 0.73) 237 
Tuttle et al. (2010) 0.62 (0.51_ 0.72) 244 
Tuttle et al. (2070) 0.63 (0.51, 0,75) 2.40 
Mac iver & Farley-Ripple (2007) 0.65 (0.38, 0,93) 1.83 
Mac iver & Farley-Ripple (2007) 0.69 (0.41, 0.96) 1.85 
Tuttle et al. (2010) 0.70 (0.56, 0,83) 2,35 
Mac \ver & Farley-Ripple (2007) 0.72 (0.45, 1,00) 1.85 
Tuttle et al. (2010) 0.78 (0.68, 0,69) 2.45 
Tuttle et al, (2010) 0.81 (0.68, 0,95) 2.32 
Hernan etal. (2011) 0.84 (0.34, 134) 140 
Subtotal (|-squared = 92.1%, p = 0,000) 0.30 (0.22, 0,37) 92.76 
RCT 
Fryer (2014) 0.13 (0.06, 0,20) 2,52 
Tuttle et al. (2075) 0.24 (0.08, 0.41) 2.26 
Angnist et a), (2010) 0.35 (0.26, 0,45) 247 
Subtotal (|-squaered = 84.7%. p = 0,001) 0.24 (0.09. 0.39) 7.24 
Overall (I-squared = 91.8%, p = 0.000) ° 0.29 (0-22, 0,36) 400,00 


NOTE; Weights are from random effects analysis 


1.76 is] 1.76 


Favors public Favors charter 
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9.1.3 Math outcomes - Year 3 


57 


Citation 


QED 

Tuttle et al. (2010) 

McDonald et al. (2008) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Teh, McCullough, & Gill (2010) 
Tuttle et al. (2010) 

Fryer (2014) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Crosby (2015) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

McDonald et al. (2008) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Herman et al. (2011) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Rose (2013) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Mac Iver & Farley-Ripple (2007) 
Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Mac Iver & Farley-Ripple (2007) 
Subtotal (l-squared = 93.2%, p = 0,000) 
RCT 

Tuttle et al. (2015) 

Tuttle et al. (2015) 

Subtotal (l-squared = 0.0%, p = 0.367) 


Overall (l-squared = 93.1%, p = 0.000) 
NOTE: Weights are from random effects analysis 


171 


Favors public 


Favors charter 


1.71 


ES (95% Cl) 


-0.03 (-0.22, 0.16) 
0.05 (-0.49, 0.58) 
0.10 (-0.05, 0.25) 
0.11 (-0.08, 0.30) 
0.13 (-0.03, 0.29) 
0.15 (0.03, 0.27) 
0.19 (0.13, 0.25) 
0.19 (0.09, 0.29) 
0.22 (0.10, 0.34) 
0.29 (0.17, 0.41) 
0.29 (0.16, 0.42) 
0.30 (0.20, 0.41) 
0.32 (0.14, 0.50) 
0.33 (0.23, 0.44) 
0.33 (0.21, 0.46) 
0.41 (-0.89, 1.71) 
0.49 (0.39, 0.60) 
0.49 (0.38, 0.61) 
0.58 (0.47, 0.69) 
0.59 (0.03, 1.14) 
0.62 (0.46, 0.78) 
0.62 (0.48, 0.76) 
0.64 (0.30, 0.98) 
0.68 (0.54, 0.81) 
0.70 (0.59, 0.80) 
0.80 (0.51, 1.08) 
0.80 (0.66, 0.93) 
0.81 (0.71, 0.91) 
0.82 (0.65, 0.99) 
0.89 (0,74, 1.03) 
1.34 (1.01, 1.67) 
0.45 (0.36, 0.55) 


0.07 (-0.09, 0.23) 
0.18 (-0,00, 0.36) 
0.12 (-0.00, 0.23) 


0.43 (0.34, 0.53) 


The Campbell Collaboration | www.campbellcollaboration.org 


100.00 


9.1.4 Math outcomes - Year 4 


Citation 


QED 

Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al, (2010) 
Tuttle et al, (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al, (2010) 
Rose (2013) 
Tuttle et al, (2010) 
Tuttle et al. (2010) 
Tuttle et al, (2010) 
Tuttle et al, (2010) 
Crosby (2015) 


McDonald et al. (2008) 


Tuttle et al. (2010) 
Tuttle et al, (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 


Mac Iver & Farley-Ripple (2007) 


Subtotal (l-squared = 93.8%, p = 0,000) 


RCT 


Abdulkadiroglu et al. (2011) 


Subtotal (l-squared = .%, p=.) 


Overall (I-squared = 93.6%, p = 0.000) 


NOTE: Weights are from random effects analysis 


-1.37 


Favors public 


Favors charter 


ES (95% Cl) 


Weight 


-0.23 (-0.38, -0.08)4.67 
-0.17 (-0.36, 0.02) 4.47 
-0.01 (-0.13, 0.11) 4.84 


0.00 (-0.12, 0.12) 
0.13 (-0.06, 0.32) 
0.13 (-0.00, 0.26) 
0.14 (0.02, 0.26) 
0.20 (-0.12, 0.53) 
0.22 (0.11, 0.33) 
0.23 (0.11, 0.36) 
0.24 (0.14, 0.35) 
0.28 (0.18, 0.38) 
0.31 (0.13, 0.49) 
0.43 (-0.26, 1.11) 
0.46 (0.36, 0.57) 
0.48 (0.36, 0.60) 
0.54 (0.41, 0.67) 
0.63 (0.53, 0.73) 
0.65 (0.54, 0.76) 
0.70 (0.59, 0.80) 
1.03 (0.69, 1.37) 
0.30 (0.18, 0.41) 


0.36 (0,29, 0.44) 
0.36 (0.29, 0.44) 


0.30 (0.19, 0.41) 


1.37 
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4.85 
4.47 
4.79 
4.87 
3.51 
4.91 
4.83 
4.92 
4.93 
4.51 
1.67 
4.92 
4.85 
4.79 
4.93 
4.90 
4.92 
3.42 
94.95 


5,05 
5.05 


100.00 


9.L5 Math outcomes KIPP vs Non-KIPP - Year 1 
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9.2 FOREST PLOTS FOR LITERACY OUTCOMES 


9.2.1 Literacy outcomes - Year 1 


Mw 
ieung et # (2009) 
Cmsbyy (2015) 

Tide ol & 


Trae at ak [UI 


Mac ive Hey feipEie (207) 


foung et a (2008) 


Young 


Young etal (200 


Vong eta (20019) 


Ten, MeCulougn, £ GIN (2010! 


foung eta (2 


o] 
Young eta (2009) 
Young etd (709) 
McDorst! etal (O00) 
trate wehae (200100) 


Vaing ew (z08) 


Mac iver 8 Mevey 
Fryer 2014) 
Yeung ot a (2007) 


Mac Ivers Far 


Puppe (2 


Satagner & Rows (2005 
Tutte et ab (2010) 
Tutie ets (2010) 


Mac iver § Partey Ripple (2097) 
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Yaing eta (205) 
Tutte et at (2010) 

fut ete 010) 
(ote & Mrper (2013 
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Yung eta (2004) 
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O97 (0.0%. 0.10) 


9.2.2 Literacy outcomes - Year 2 


Citation 


QED 

Young et al. (2009) 

Young et al. (2003) 

Young et al. (2009) 

McDonald et al. (2008) 

Tuttle et al, (2010) 

Teh, McCullough. & Gill (2010) 
Young et al. (2003) 

Young et al. (2009) 

Tuttle et al, (2010) 

Tuttle et al, (2010) 

Young et al, (2009) 

Tuttle et al, (2010) 

Fryer (2014) 

Tuttle et al, (2010) 

Tuttle et al. (2010) 

Tuttle et al, (2010) 

Crosby (2015) 

Mac |ver & Farley-Ripple (2007) 
Fryer (2014) 

Hernan et al. (2011) 

Henman et al. (2011) 

Tuttle et al, (2010) 

Young et al. (2009) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al, (2015) 

Tuttle et al. (2010) 

Tuttle et al, (2010) 

Tuttle et al. (2010) 

Rose (2013) 

McDonald et al. (2008) 

Tuttle et al, (2070) 

Tuttle et ai, (2070) 

Tuttle et al, (2010) 

Tuttle et al, (2010) 

Mac Iver & Farley-Ripple (2007) 
Tuttle et ai, (2010) 

Tuttle et al, (2010) 

Tuttle et al, (2010) 

Tuttle et al (2010) 

Gallagher & Ross (2005) 
Young et al. (2003) 

McDonald et al, (2008) 

Tuttle et al, (2010) 

Mac Iver & Fartey-Ripple (2007) 
Subtotal (l-squared = 80.8%, p = 0.000) 


RCT 

Fryer (2014) 

Angrist et al. (2010) 

Tuttle et al (2015) 

Subtotal (l-squared = 0.0%, p = 0.384) 


Overall (l-squared = 79.6%, p = 0,000) 


ES (95% Cl) 


-0.22 (-0.41, -0.04) 
-0. 17 (-0,30, -0,05) 
-0.17 (-0,51, 0.16) 
-0. 12 (-1.40, 1.16) 
-0.12 (-0.24, 0.00) 
-0.10 (-0.26. 0.06) 
-0.08 (-0,33, 0.17) 
-0,06 (-0,23, 0.11) 
-0.04 (-0, 19, 0.11) 
-0.03 (-0. 16, 0.10) 
-0.02 (-0,26, 0,22) 
-0.02 (-0, 12, 0.08) 
-0.00 (-0,04, 0.03) 
-0.00 (-0 19, 0.18) 
0.00 (-0.10, 0.10) 
0.02 (-0.17, 0,21) 
0.06 (-0.12, 0,24) 
0.06 (-0,21, 0.33) 
0.07 (0,01. 0.14) 
0.10 (-0.15, 0.35) 
0.13-(-0.01, 0.27) 
0.13 (0,02, 0.24) 
0.14 (-0,21, 0,50) 
0.16 (0,00, 0.32) 
0.17 (0,05, 0.29) 
0.18 (0,08, 0.27) 
0,20(0,08, 0.32) 
0.21 (0,08, 0.34) 
0.21 (0,08, 0.33) 
0.21 (-0,12, 0.54) 
0.21 (-0.77. 1.20) 
0.22 (0, 12, 0.32) 
0.22 (0,08, 0.36) 
0.24 (0.13, 0.35) 
0.24 (0,11, 0.37) 
0.28 (0.01. 0.55) 
0.29 (0, 19, 0.40) 
0.34 (0,15, 0.48) 
0.33 (0,23, 0.44) 
0.34 (0,23, 0.46) 
0.37 (-0.07, 0.81) 
0.38 (0,05, 0,71) 
0,39(-0,13, 0.91) 
0.44 (0,30, 0.58) 
0.44(0. 17,071) 
0.12(0,07, 0.17) 


0.07 (-0.00, 0.14) 
0.12 (0,03, 0.21) 
0.18 (0,02, 0.35) 
0.10/0.04. 0.15) 


0.12 (0,08, 0.16) 


271 


157 


2.66 


236 


276 


257 
1.14 
O19 
272 
242 
2.66 
2.48 
143 
2.68 
2.21 
2.69 
257 
O77 
1.15 
0,59 
233 
1.45 
92,12 


292 
276 
220 
7.88 


100.00 


NOTE: Weights are from random effects analysis 
-14 ia) 14 


Favors public Favors charter 
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9.2.3 Literacy outcomes - Year 3 


Citation 


QED 

McDonald et al. (2008) 

Herman et al. (2011) 

Tuttle et al. (2010) 

Teh, McCullough, & Gill (2010) 
Tuttle et al. (2010) 

Tuttle et al, (2010) 

Fryer (2014) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al, (2010) 

Rose (2013) 

Tuttle et al, (2010) 

Crosby (2015) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al, (2010) 

Tuttle et al, (2010) 

Tuttle et al, (2010) 

Tuttle et al, (2010) 

Tuttle et al, (2010) 

McDonald et al. (2008) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Tuttle et al. (2010) 

Mac Iver & Farley-Ripple (2007) 
Tuttle et al, (2010) 

Tuttle et al, (2010) 

Mac Iver & Farley-Ripple (2007) 
Tuttle et al, (2010) 

Subtotal (l-squared = 90.4%, p = 0.000) 
RCT 

Tuttle et al, (2015) 

Tuttle et al, (2015) 

Subtotal (l-squared = 0.0%, p = 0.431) 


Overall (l-squared = 89.8%, p = 0.000) 
NOTE: Weights are from random effects analysis 


-1.6 


Favors public 


Favors charter 


1.6 


ES (95% Cl) 


-0.22 (-0.76, 0.31) 
-0.114 (-0.40, 0.17) 
-0.114 (-0.26, 0.04) 
-0.10 (-0.26, 0.06) 
-0.08 (-0.20, 0.04) 
-0.00 (-0.11, 0.10) 
0.00 (-0.06, 0.06) 
0.02 (-0.08, 0.12) 
0.08 (-0.11, 0.27) 
0.12 (-0.07, 0.31) 
0.12 (-0.01, 0.25) 
0.14 (0.02, 0.26) 
0.16 (-0.17, 0.49) 
0.18 (0.06, 0.30) 
0.20 (0.02, 0.37) 
0.24 (0.08, 0.40) 
0.24 (0.11, 0.37) 
0.28 (0.16, 0.41) 
0.28 (0.18, 0.39) 
0.28 (0.18, 0.39) 
0.28 (0.18, 0.38) 
0.29 (0.19, 0.40) 
0.30 (-0.99, 1.60) 
0.33 (0.22, 0.45) 
0.39 (0.25, 0.53) 
0.40 (0.27, 0.53) 
0.43 (0.15, 0.71) 
0.43 (0.33, 0,53) 
0.43 (0.27, 0,59) 
0.44 (0.12, 0,76) 
4.02 (0.87, 1.17) 
0.21 (0.13, 0.29) 


0.14 (-0.04, 0.32) 
0.24 (0.08, 0.39) 
0.20 (0.08, 0.31) 


0.21 (0.14, 0.29) 


62 The Campbell Collaboration | www.campbellcollaboration.org 


100.00 


9.2.4 Literacy outcomes - Year 4 


Citation 


QED 

Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Rose (2013) 
Tuttle et al, (2010) 
Crosby (2015) 


Mac |ver & Farley-Ripple (2007) 


Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
Tuttle et al. (2010) 
McDonald et al. (2008) 


Subtotal (l-squared = 81.4%, p = 0.000) 


RCT 


Abdulkadiroglu et al. (2011) 
Subtotal (|-squared = .%, p =.) 


Overall (l-squared = 80.5%, p = 0.000) 


NOTE: Weights are from random effects analysis 
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Favors charter 


The Campbell Collaboration | www.campbellcollaboration.org 


% 
ES (95% Cl) 


-0.15 (-0.30, 0.00)4,54 
-0.12 (-0.24, 0.00)5.07 
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10 Appendix I 


10.1CODING MANUAL 


Study Characteristics 


Study ID 

Number the study to follow sequentially after the last study ID. If multiple studies are reported 
in the same article/report, follow the study ID given to the article with separate letters (e.g., A, 
B, C, etc.) to indicate the separate studies reported. 


Report ID 
Separate articles/reports that provide information on the same study (i.e., the same sample 
appears in multiple articles or reports) will have the same study ID, but separate report IDs. 


Author(s) and (year) 
Specify the author(s) and year (in parentheses) as they would appear at the start of an APA 
citation. This text will be used for final forest plots. 


Publication type 
1 = journal article, 2 = dissertation/thesis, 3 = book/book chapter, 4 = report, 5 = other 


Year published 
Indicate in YYYY format. 


Country published 
1 = United States, 2 = United Kingdom, 3 = other 


Design and Charter Characteristics 


ie 
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Assignment procedure 

1 = random, 2 = matched, 3 = convenience sample, 4 = other, 5 = unknown 

Note: For this type of study, random assignment will typically be through a lottery process. To 
qualify for matched assignment, individuals must be matched on baseline achievement 
measures. If convenience sampling is used, the study must still report baseline achievement 
data, as well as demographic data; the researcher might not have chosen a comparison group 
that matches as closely as possible, however. 
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8. Charter explicitly “no excuses”? 
1=yes, 2=no 
Note: Choose “yes” if either A) the study describes the charter school as ‘No Excuses’, B) the 
study describes a known ‘No Excuses’ brand of charter school, or C) the coder can identify (by 
online sources or otherwise) that the charter school is a ‘No Excuses’ brand of charter. 
Otherwise, the study must report (or reference sources must verify), that the charter school 
meets all five criteria of the ‘No Excuses’ model: 1) data-driven policies, 2) expectations of 
college attendance, 3) restrictive discipline policies, 4) increased instructional time, 5) intensive 
teacher training. Leniency should be given for operationalization (e.g., what is considered 
“restrictive” discipline policies), but each criterion should be clearly referenced. 


9. Number of years charter has been active: 
Note: This applies to the specific school (i.e., not the brand of the school, if applicable). Code - 


999 if cannot tell. 


10. Charter school type: 
1 = managed by a charter management organization, 2 = stand-alone charter school 


11. Comparison school type 
1 = general public school (or no indication otherwise), 2 = no excuses public school, 3 = mix of 


public schools, charter schools, and/or private schools 


Baseline Characteristics 

Note: Code characteristics as percentages (i.e., ##%) rather than proportions when appropriate. If 
the charter and comparison samples are matched on certain characteristics (e.g., prior 
achievement, ethnicity, etc), and only the pooled characteristics are reported, code those pooled 
characteristics for both samples and code the other characteristics as cannot tell. If cannot tell, 
code as -999. 


12. Charter grade 

13. Comparison grade 
Note: Code average grade to nearest whole number. If international (or unconventional) grade 
system is used, use average age and subtract 5.5 from average age for grade estimate. 

14. Charter percent minority 

15. Comparison percent minority 

16. Charter percent free/reduced lunch 

17. Comparison percent free/reduced lunch 

18. Charter percent male 

19. Comparison percent male 

20. Charter school size 


21. Comparison school size 
22. Charter class size 


23. Comparison class size 
Note: Code student/teacher ratio if given. 
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Assessment Details 


24. 


25. 


26. 


Test name 


Give short text name for assessment 
Assessment content 


1 = math, 2 = literacy, 3 = both 
Assessment scope 
1 = state, 2 = nation, 3 = other 


Effect Size 


27. 
28. 
29. 
30. 
31. 


32. 
33. 
34. 
35. 
36. 


37. 
38. 
39. 
40. 


41 


45. 
46. 
47. 
48. 
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Charter sample size at assignment 

Comparison sample size at assignment 

Charter sample size at assessment 

Comparison sample size at assessment 

Years between assignment and assessment 

Note: Round to nearest year (e.g., 9 months post-assignment = 1 year). If multiple follow-up 
years are reported, report on all years (with appropriate assessment sample size and effect 
size information associated with each year). 

Charter math DV mean 

Charter math DV standard deviation 

Comparison math DV mean 

Comparison math DV standard deviation 

Charter math DV percent passed 

Note: If information is given for DV mean/standard deviation and percent passed, use 
mean/standard deviation. 

Comparison math DV percent passed 

Charter other math effect size measure 

Comparison other math effect size measure 

Charter literacy DV mean 


. Charter literacy DV standard deviation 
A2. 


43. 
44. 


Comparison literacy DV mean 

Comparison literacy DV standard deviation 
Charter literacy DV percent passed 

Note: If information is given for DV mean/standard deviation and percent passed, use 
mean/standard deviation. 

Comparison literacy DV percent passed 
Charter other literacy effect size measure 
Comparison other literacy effect size measure 
Estimation used for ES? 

1=yes, 2=no 

Note: If ‘yes’, explain what estimation is used. 
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CampbellCollaboration 


About this review 


The No Excuses charter school model focuses heavily on high academic expectations, rigid and 
consistent discipline, extended instructional time, intensive teacher training, and increased 
parental involvement. 


This review examines the effect of No Excuses charter schools on math and literacy 
achievement. On average, No Excuses charter schools are associated with greater student 
gains on standardized measures of math and literacy achievement when compared to 
traditional public schools—with higher gains for math. 


WSs 
S 
SS 


N\\ 
\\ 


ee ge 
a |] 


The Campbell Collaboration Mailing address: Visiting address: Website: 
info@campbellcollaboration.org P.O. Box 4404, Nydalen Pilestredet Park 7 www.campbellcollaboration.org 
Phone: (+47) 23 25 50 00 N-0403 Oslo, Norway (Entrance from Stensberggata) 


